amadan
Merge branch 'design-maintenance-exec' into main
1 file changed,
+113
−0
implementation/maintenance-exec.md+113 −0
CI failed — run details
diff --git a/implementation/maintenance-exec.md b/implementation/maintenance-exec.md| new file mode 100644 |
| index 000000000..44d86fa39 |
| --- /dev/null |
| +++ b/implementation/maintenance-exec.md |
| @@ -0,0 +1,113 @@ |
| +# Fenced maintenance for an existing exec instance |
| + |
| +Status: revised after independent Claude Fable R3 CHANGES REQUIRED at0c59ea2. R1 quota and R2 permission failures had no verdict. This revision addresses the four source-backed blockers and two coordinator source-audit gaps; final design approval remains pending. Platform discussion117/task14; base dcf0cad333405aa8990af62c0871bd6ef63d96ba. This is an implementation proposal, not authorization to run a maintenance operation. User authorization covers implementing the missing supported platform command as part of Oficina readiness. Concrete deployment, fresh owner approval and operation receipts remain required. |
| + |
| +## Source and scope |
| + |
| +Read the authoritative infra snapshot259913017c2aeb99554e2c1384c2117d279f6e2a: instance provisioning2026-08-04, exec config delivery2026-08-02, route-scoped config2026-08-16, exec accounting2026-08-07, SQLite corruption defence2026-08-15 and scheduled work2026-08-23. The current platform instance-move spec2026-08-28 supplies related lifecycle ordering, not permission to reuse move/cut: that operation later purges files and changes residence. Actual source at dcf0cad is authoritative for implementation seams. |
| + |
| +The first supported class is a provisioned, single-residence, hibernating, activator-managed exec instance with one canonical primary host and an existing local database. Refuse fleet placement, pools, unit/addr/static/service/backing/tunnel routes, non-hibernating instances, unknown shape, non-owned registry rows and an in-flight move. Aliases resolve to an explicitly displayed primary before approval; they never widen the operation to another database. No cross-box execution takeover, missing-DB restore, new DB creation, live seed, raw SQL, host shell or root command in this slice. Other shapes return unsupported without changing state. Prove current Home roles meet these checks through the new supported preflight; restart's asleep report is insufficient. |
| + |
| +The operation uses a shipped, verified release of the selected app, pinned by full version and artifact digest. It executes that artifact directly with a bounded argv array, never a shell, live symlink or caller-supplied executable path. The platform supplies the exact existing DB through its defined private command contract; the app owns schema/semantic validation, transactions and idempotent operation receipts. This is existing instance authority under the platform's accepted shared-uid exec isolation model, not a claim to sandbox a hostile app binary from sibling files. It grants no root or deployment credentials. Home's owner receipt package and Docs docs-seal remain separate reviewed consumers. |
| + |
| +## Owner authorization and command interface |
| + |
| +Add `carlos maintenance prepare`, `status`, `result`, `retry` and `resume` under one documented command. All account/app/primary-host selectors are mandatory and exact; no default wildcard or account-wide variant. Prepare validates a JSON input file (0600, regular file, bounded), uploads it over the authenticated console API, and returns an immutable request ID plus a console review URL. CLI credentials are isolated in tests through carlosTestEnv and scratch cwd. Public status/result metadata never echoes argv, private input/output, secrets or app stderr. |
| + |
| +An ordinary CLI principal has no authentication-time proof (svcprincipal.Principal). Do not treat a recently minted bearer, service minter address or app-token instanceGate as fresh owner authority. Only a current human owner can prepare/read these operations. The console approval page shows exact primary host/account/app, operation label, artifact version/digest, configuration binding and downtime consequence. Its mutation gates run session -> CSRF -> current owner byte-identical404 -> existing requireFreshAuth last. Approval binds the immutable request digest, selected instance record revision and preflight revision; a changed request requires a fresh approval. The maintenance handler uses that canonical order. The invariant is current-owner non-disclosure before any redirect and fresh-auth strictly last; existing handlers with owner before CSRF are not proof that session or CSRF can be skipped. The existing Keymail step-up supplies the actual gesture. No agent fabricates it. |
| + |
| +Prepare is inert until this browser approval. Approval is one-time and expires before a queued request can start indefinitely: five minutes from the actual owner gesture, checked before accepting the fence request. Once a matching accepted fence exists, recovery may continue stopping/checkpointing safely without another gesture; starting a new command attempt, changed inputs, retry and resume each require their own current-owner fresh approval. A lost CLI connection does not cancel, rerun or resume anything. Status polls immutable receipts; command attempts have separate IDs and never arise from GET or polling. |
| + |
| +The public request record holds random request/attempt IDs, binding digests, bounded operation label, timestamps and state only. Encrypt argv and private input immediately using the existing deployment secret infrastructure with a NEW purpose-separated authenticated envelope bound to account/app/host/request/attempt/direction and envelope version. Do not reuse an existing config envelope with different semantics. Public binding hashes cover randomized ciphertext, not low-entropy plaintext. Reject absent/decrypt-failed keys; do not log decrypted fields. Input is capped at64KiB, argv at32 entries/8KiB total, execution at five minutes by default with a hard ten-minute ceiling. Private result is capped at64KiB. Stdout is the private result channel, captured into a bounded private buffer and encrypted before durable publication; stderr is discarded. Neither stream uses teeStd, journald or app logs. The command gets the private JSON request only on stdin. Result retrieval requires current human-owner authority and writes only to an explicit exclusive0600 local output file, never stdout. No retrieval capability is exposed in a public receipt. Expired private payloads are deleted after24h; nonsecret receipts and ciphertext hashes remain for audit. A result lost before durable publication is ambiguous, not a success with an invented empty response. |
| + |
| +## Durable state and all consumers |
| + |
| +Use a durable guard at control/maintenance/hosts/<primary-host>/guard.json as the denial authority, updated by conditional writes. It survives ordinary apps/<account>/<app>/ deletion and pins the original account/app/record identity. The key is global by canonical primary hostname because both DefaultDBPath and HibernatePrefix derive storage identity from host alone. A replacement record under another account/app must encounter the same guard and refuse the identity mismatch; it must not look up a fresh account-scoped key and reuse the old warm database or replica. This slice never reassigns a guarded hostname to another identity. Receipts and attempts live under the same hostname namespace with their original account/app authenticated in every envelope. Add an optional maintenance reference on provision.Record as a CAS-bound projection and conflict guard, not the sole barrier to a wake. The guard points to immutable request/attempt envelopes and records a monotonically increasing fence generation, primary host, owning box identity, expected record/config/artifact binding and phase. Permanent operation receipts live beside the guard, outside the app-purge prefix; completing one does not erase its generation or history. No unbounded list is embedded in the instance record. Old records with no maintenance history retain their existing behavior. |
| + |
| +Phases are awaiting-approval (console-private intent, not yet a route fence), requested, quiescing, fenced, executing, succeeded, failed, ambiguous, resume-requested and resumed. requested through resume-requested all deny serving. A successful command stays fenced. Terminal failure never auto-clears. `executing` is durably committed before spawning; after restart an executing attempt becomes ambiguous and is NEVER automatically repeated. CAS binds phase+attempt+generation+owner; stale workers cannot overwrite new attempts or clear a later fence. Control serial nudges convergence, never supplies authority. |
| + |
| +Inventory every parser and writer of Record, not only the new handler: provision Get/List/Create/update/delete/shape/channel/environment/region/move, console account/app deletion and restore, root adoption, edge registry reconstruction, restart workers, config sync, scheduling and retention. All upgraded mutation writers preserve the additive object and reject conflicting move/delete/shape/environment/channel actions while fenced unless they are the separately authorized resume transition. App-wide release promotion may proceed for siblings but cannot change this instance's pinned executable or environment. No config sync/removal/reap or adoption symlink change may affect a fenced instance. The implementation must pin each audited consumer in tests; an incomplete inventory blocks implementation approval. |
| + |
| +At boot, reconstruct the local durable fence from authoritative records BEFORE enabling HTTP, schedule or sidecar wake. Unknown phase, failed record read, missing active receipt or unsupported maintenance version fails closed for that instance. Enforce admission inside activator begin/wake and sidecar/scheduled paths, not just edge Status. A caller may hold a stale ready Route; after acquiring the normal per-host lease it must re-read/check maintenance authority before starting the child. This closes absence-read -> maintenance-request -> lease-acquire races. Edge returns fixed503/no-store for a fenced route, including aliases; timers remain recorded but cannot wake it. Unknown/missing instance authority never becomes a fresh seed fallback. |
| + |
| +Only upgraded maintenance-aware agents may serve an instance once maintenance is enabled for it. Deployment preflight verifies the capability and version of every eligible residence/serving box, active primary, console binary and every principal/binary that can rewrite a provisioned record or execute app lifecycle changes, together with their enforced rollback floors. Pre-floor writers do NOT preserve the projection: current GetRecordETag/UpdateRecord and updateRecord unmarshal into Record then remarshal, silently dropping unknown fields. A missing projection under an active or resumed guard is a mismatched-record denial, never an ordinary legacy record. An old console is therefore no more a safe rollback than an old serving agent; checking only box versions is insufficient. An uncontrolled record-writing principal makes the maintenance capability unavailable. An old platform binary that ignores the new record is NOT a safe rollback while a fence exists. This is an explicit operational release floor, not a property an older parser can enforce. Failover can observe the fence but cannot execute on a different box or recreate a missing local DB. Loss of the pinned box leaves the operation fenced and unavailable until a separately reviewed recovery path exists. |
| + |
| +## Exclusive execution boundary |
| + |
| +The worker first verifies its current ownership of the exact provisioned primary route, supported class and request/config/artifact binding. Region equality and Quiesced alone are insufficient: Quiesced only proves the local activator map has no entry. It writes the durable fence, marks local admission closed and retires the child/sidecar. It must obtain explicit success evidence that both are stopped and the old replication store is closed; current stop logs and deletes the entry even after Stop/Close errors, so ordinary RetireExcept+Quiesced cannot be treated as that proof. Add a checked stop result rather than weakening the existing routine stop semantics. |
| + |
| +Acquire the existing per-host replica lease only after the above closure, and retain it continuously through pre-checkpoint, command execution and post-checkpoint. A lease held elsewhere means wait/refuse, never force-take. Add a narrow lease checkpoint primitive that opens/syncs/closes a temporary replication store while the SAME Held renewer continues to own the lease; it neither releases nor creates a second concurrent replica. Reuse the existing replica-store construction and SyncAndWait implementation. Do not use Replicator.Close (which releases leadership) between these phases. The primitive checks ownership before/after, surfaces close/sync failure and cannot operate on a spent Held. Its deterministic tests prove no acquisition gap and one replica at a time. |
| + |
| +Under that lease, validate the exact platform-derived existing DB path: regular file, no symlink substitution, within the selected DataDir namespace, expected device/inode/file identity from the stopped preflight, expected route DB path and no quarantine/corruption state. Refuse missing/changed files without Restore, Seed, create, unlink or replacement. A checked pre-command checkpoint proves the retained replica carries the stopped tail. Record a nonsecret DB identity/checkpoint receipt, not document contents. The app receives only the existing DB path; its command must itself refuse creation and validate any key fingerprint/semantic preconditions before mutation. |
| + |
| +Run the pinned artifact under the existing selected instance cgroup/tier and approved merged environment snapshot, stripped of platform-only/control/purge variables exactly as the established exec boundary requires. Do not call EnsureAdminToken or silently create config during maintenance. Resolve and verify the artifact before execution without depending on the mutable live link. Capture an immutable encrypted environment binding during approved preflight, not a plaintext public hash; worker verifies it again before spawn. Concurrent environment/instance overlay changes fail the attempt instead of silently switching keys. |
| + |
| +Persist executing before spawn. Bound the whole process tree and private output; timeout, output overflow, lost lease, cancellation, parent death or unexpected descendant forces termination and an explicit failed/ambiguous receipt. Use cgroup-wide stop/emptiness proof plus normal Wait; PID alone is not death evidence. Monitor lease health with a deadline before lease expiry, not just eventual error polling. Parent/agent crash must not leave an orphan writer eligible to run after a new lease is acquired. Platform cgroup/parent-death behavior must be verified or extended before this path is allowed; unavailable proof leaves it fenced. No result exit0 can substitute for checked empty process tree. |
| + |
| +After a successful app exit, revalidate the expected DB identity policy (in-place SQLite VACUUM is allowed only if the reviewed operation's identity rule explicitly permits its file behavior), run checked integrity validation and a final checkpoint under the retained lease, then atomically publish encrypted private result plus succeeded receipt referencing its digest. For VACUUM or any permitted whole-file rewrite, success additionally requires a restore-verification receipt after the final checkpoint and before success publication, while the same lease remains held and the source is stopped. Use the existing VerifyReplica/CheckDB shape against an exclusive operation-private scratch directory, never the live path. Extend that shape with a read-only verification callback before its cleanup so the approved consumer can compare the restored database's post-operation semantic receipt/digest with the stopped source; CheckDB and the 32-bit change counter alone cannot prove freshness. Do not emit document contents or plaintext digests publicly; bind sensitive comparison material into the encrypted result. Missing replica, corrupt restore, stale semantic receipt, timeout or cleanup failure keeps the operation failed/fenced. Remove the private scratch DB/WAL/SHM on every outcome without touching the live DB or historical replicas. A real litestream v0.5.14 full-write/VACUUM/checkpoint/restore test must prove post-rewrite data and receipt fidelity, with stale and corrupt replica negative controls. The existing local inode probe is explicitly not this evidence. |
| + |
| +Publication failure is ambiguous. Failure paths may preserve a partially mutated DB; never auto-restore, delete historical replicas or claim rollback. Release the lease only when command/replica shutdown is proven and the durable fence still denies every wake; otherwise keep/abandon leadership conservatively and record ambiguity. No command reruns automatically after crash. |
| + |
| +## Retry and resume |
| + |
| +Retry creates a new immutable attempt with a fresh owner approval and the previous attempt receipt bound into its request. The app must supply idempotent operation semantics or explicitly describe repair; platform cannot infer that a failed command made no changes. Before retry, re-prove stopped processes, current record generation, exact DB/artifact/config and checkpoint. The same request ID with changed bytes conflicts. Reading a finished receipt never executes anything. |
| + |
| +Resume is a separately approved operation, not `clear maintenance`. It names an expected serving release/artifact, configuration/key binding and successful app-specific compatibility-check command from that release. Platform verifies all pins under the fence/lease, runs the non-serving checker against the existing DB, verifies integrity/checkpoint and records its result. The app checker owns format/key/authorization compatibility; platform cannot infer it from exit of an earlier migration or a byte-compatible old binary. Docs TeamMode's minimum authorization-aware release floor is one such consumer requirement. |
| + |
| +Only after the checker succeeds may the worker publish resume-ready evidence, install the pinned serving release/config and arrange the lease-to-serving replication handoff. Clear the fence through one guarded transition bound to this evidence; old/new worker races cannot resurrect a superseded resume. Admit HTTP only after the exact new process and replication are healthy and report the approved version; retain a durable resumed receipt. If process startup/replication/version verification fails, return to fenced/failed, not ready. The implementation must decide and test the precise lease handoff before code approval; simply releasing and hoping the next wake uses the approved pins is insufficient. |
| + |
| +## Concrete implementation seams for the next review |
| + |
| +These decisions replace the corresponding open mechanism questions above; the complete consumer inventory and final resume transition still need to be checked against implementation source before code approval. |
| + |
| +**Private transport does not require a console decryption key.** Actual appconfig/crypto.go deliberately puts CARLOS_SECRETS_PUBKEY on the encrypting side and CARLOS_SECRETS_KEY on decrypting hosts. The console must not gain the private key for maintenance. Add a shared purpose-aware envelope primitive in internal/maintenance, reusing the established ECDH-P256/HKDF-SHA256/AES-GCM shape but a new HKDF info string `carlos-maintenance-v1`; authenticated data includes canonical length-prefixed version, direction, account, app, primary host, request ID, attempt ID and recipient fingerprint. Generate a fresh ephemeral key and nonce per envelope. Unknown version/purpose/direction/recipient or changed binding refuses before plaintext is used. Golden vectors and cross-purpose mutation tests are required; old appconfig Seal/Open remain byte-compatible. |
| + |
| +The preparing CLI generates a per-request P-256 result key pair and saves its private key in its exclusive0600 local pending-operation file before upload. The owner's reviewed request includes the public recipient fingerprint. Request payload encrypts to the existing deployment secrets public key; the worker decrypts it with the existing host key. Private stdout encrypts to the CLI result public key. Console and bucket persist only ciphertext and authenticated metadata; the console can authorize delivery but cannot decrypt results. The CLI requires both a current human-owner API session and the saved recipient private key to produce the private result file. Lost local result keys make that result unavailable; no automatic replay or server-side plaintext fallback. Any operator-level recovery of an idempotent receipt is a new, approved request. Pending local keys and payloads are outside repository/terminal logs; cleanup does not silently destroy the only copy before the caller durably saves a result. |
| + |
| +**Environment pinning is private.** Public approval shows the selected named/default config revisions and a randomized snapshot-envelope digest, never a plaintext secret digest. A worker preflight captures the exact merged instanceEnv map into an authenticated encrypted snapshot, retaining both config source revisions and the box-local overlay identity. The owner approval binds that snapshot ciphertext and record revision. Before execution the worker compares the decrypted approved map with the current merged map in memory and refuses changes; it does not reload a newer key into the command. The snapshot is the command's environment. No EnsureAdminToken side effect or restoration of a missing overlay occurs. Adoption/config writers must preserve the pinned snapshot while the fence is active; checking only manifest generation omits local overlay changes and is insufficient. |
| + |
| +**Checked stop is a distinct result, not a synonym for Quiesced.** Add a per-host maintenance fence map inside activator.Service, protected by the same mutex that inserts waking entries. `FenceAndStop(ctx, host, generation)` sets the local admission barrier before inspecting a waking/awake/stopping entry, then waits for that exact entry's terminal result. Every path through begin, ordinary serve, scheduled wake, BeginSidecarRequest and the sweepOnce sidecar-adoption/restart branch consults the barrier, including stale Route callers. The existing serve path checks Kind/Hibernate without the sweeper's Status==ready condition; registry status is therefore not the barrier. Fence installation, entry-state transition and sidecar replacement are serialized under the same mutex. A deterministic fence-versus-dead-sidecar sweep test must prove no replacement is spawned after the barrier and checked stop evidence covers the exact final child and sidecar identities. Extend the internal stop result to record sidecar Stop/Exited, main Stop/Exited and replication Close outcomes before removing the entry; ordinary stop logging behavior stays compatible, but the maintenance caller refuses on any missing/failed evidence. A missing entry is only part of the proof: strict cgroup emptiness and absence of an active replication owner must also hold. On agent restart an outstanding executing attempt is ambiguous, and checked cleanup is required before any retry/resume. CAS request generations prevent an old local completion from clearing a new barrier. |
| + |
| +**Checkpointing retains the same lease.** Add `(*lease.Held).Checkpoint(ctx, dbPath) (CheckpointReceipt, error)` as an exclusive, non-consuming operation. It uses an extracted common replica-store constructor, opens the single temporary store, calls SyncAndWait, closes it with checked errors and returns while the Held renewer remains active. No ReleaseLease occurs on success or failure. Checkpoint cannot run concurrently with itself, Replicate or Release, or on a spent handle; use an explicit ownership/state guard rather than relying on caller memory. Receipt is emitted only after sync AND store close AND final lease-health check. A failure leaves the handle owned but unusable for command execution until the worker records and resolves the failure. The eventual serving `Replicate` transfer retains its existing ownership semantics; existing callers are not silently changed to keep leases forever. |
| + |
| +Expose a conservative monotonic safe-until deadline from lease acquisition/renewal, captured before the corresponding remote request and shortened by a fixed termination margin. Maintenance must stop and prove its process tree empty before that deadline. A renewal failure or elapsed deadline signals immediate termination even if the wall clock moves backwards. Tests use shortened TTLs and a blocked renewal to prove termination precedes takeover. No new lease acquisition can be reported as proof an old process died. |
| + |
| +**Maintenance requires strict process containment.** Current cgroup.Manager.Start deliberately falls back to spawn-then-move and even an unaccounted spawn; its source notes grandchildren can remain outside the host cgroup. That fallback is acceptable accounting behavior but is not the maintenance boundary. Add a separate strict start path requiring Linux cgroup v2, CLONE_INTO_CGROUP, a writable delegated host subtree and usable cgroup.kill/events. It never falls back or starts before containment is established. A missing capability means unsupported, not a less safe command. Use the existing tier limits; failed limit/containment setup refuses. Kernel cgroup.events populated=0 is required after termination; PID/Wait alone does not cover descendants. Use cgroup.kill for the complete subtree on timeout/lease loss and verify emptiness within the remaining safe lease window. Keep ordinary Start behavior unchanged. |
| + |
| +The maintenance capability preflight also requires the platform-managed agent unit to explicitly enforce KillMode=control-group, SendSIGKILL=yes and a bounded TimeoutStopSec. The checked effective unit, not an assumed systemd default, is part of the deployed capability receipt. Add parent-death protection on the direct command as a new requirement; current production spawn code does not supply it. It is not claimed to cover grandchildren. Agent main-process death must cause systemd to kill the entire delegated subtree before restart; test actual systemd behavior on the isolated Linux platform canary, including a descendant holding the output pipe. Without that deployed proof this class remains disabled. OOM/forced-kill and agent-restart tests must prove no surviving writer is mistaken for a clean stop. This is containment against process lifecycle failure under the existing trusted app/shared-uid model, not protection against a malicious instance intentionally migrating out of its delegated cgroup. |
| + |
| +**Resume must use the retained lease and a scoped internal permit.** The proposed seam is `Service.ResumeFenced` taking an already-held lease and a worker-created opaque permit bound to record/attempt/generation, approved serving artifact and environment snapshot. Ordinary callers cannot construct the permit. It starts replication and the exact pinned serving process while all public admission remains fenced; no Restore, Seed or live-link resolution is performed. It checks health and the app's actual reported version before the worker CASes the matching durable resume outcome. Failed CAS, version mismatch or process/replication failure invokes checked stop and retains the durable fence. The proposed durable serving contract below closes the concurrent-promotion and durable-clear/local-ready race; a bare release-and-next-wake or transient local pin is rejected. Its full integration remains subject to the final review. |
| + |
| +## Durable guard and serving compatibility proposal |
| + |
| +The source audit found two reasons a record field alone is insufficient. provision.DeleteRecord currently performs an unconditional Store.Delete, and console/ownership.go purges the app prefix. A prior read followed by that delete can erase a newly written record fence. Therefore every maintenance-aware primary wake checks the outside-app guard, including after lease acquisition and before any Restore/Seed. A missing/malformed/mismatched record while a guard is active denies admission; a recreated record with the same hostname cannot claim the old database. Guard creation is create-only for the first operation and CAS on the monotonic generation thereafter. The worker may proceed only when the matching reference is also CAS-installed on the expected record revision. Failure between guard creation and record projection remains fenced/failed; it never clears the guard to repair availability automatically. Denial requires either barrier; execution/resume authority requires both matching barriers plus the approved request. This is a deliberate fail-closed multi-object protocol, not a claim to cross-object atomicity. |
| + |
| +Extend the store's conditional operation support with DeleteIfMatch, including directory/S3/prefix implementations. The pinned AWS S3 SDKv1.105.0 already exposes DeleteObjectInput.IfMatch. GetRecord with its etag, active-guard refusal and conditional delete close the old read/check/delete race for every instance delete caller. Unsupported store implementations refuse this operation; never fall back to unconditional delete. The backend capability proof must attempt deletion with a wrong ETag against a disposable object and verify a conflict with the object retained; merely exposing the SDK field does not prove a compatible server enforces it. App purge and automated trash sweeps must check outside-app guards before ANY destructive step and must route instance-record deletion through this guarded primitive. Maintenance may start only on an active, untrashed app, and rechecks that lifecycle state after installing its durable guard and before executing. A concurrently removed/trashed/replaced app makes the operation fail closed; the external guard and receipts survive. Tests must interleave trash/purge/restore/guard creation at these boundaries; merely checking a list of guards once is not sufficient evidence that the race is closed. |
| + |
| +Make that lifecycle exclusion concrete with a separate durable CAS object at control/maintenance/apps/<account>/<app>/lifecycle.json. All upgraded TrashApp, RestoreApp and PurgeApp callers (including unattended reconciliation) acquire an exclusive operation claim there before their first mutation or destruction. Absence is acquired by create-only Put; otherwise CAS the observed idle generation to an immutable operation ID and kind. A maintenance fence likewise acquires the claim before creating/changing its hostname guard, then rechecks the active, untrashed app and exact record. A claimant that lost CAS performs no side effect. This common create/CAS point closes the two simultaneous absence-read race; a guard-list precheck alone does not. |
| + |
| +Claims are not expiring locks: crash or partial purge retains an incomplete claim and blocks competing lifecycle/maintenance operations until that specific idempotent operation is explicitly recovered. Never reclaim by wall-clock timeout. Successful ordinary lifecycle operations CAS their matching claim back to idle at a higher generation; retain history instead of deleting the object and allowing ABA. Maintenance holds its app lifecycle claim until both resumed references and serving health agree; a failure or orphan hostname guard leaves the claim held. This serializes maintenance attempts within an app, but does not block compatible sibling release promotion or ordinary serving. A crash after claim acquisition but before hostname guard creation has not authorized execution; current serving may continue until the actual host fence is installed, while destructive lifecycle operations remain excluded. Recovery requires the matching operation/owner authorization and never creates or reruns a command from a bare claim. Crash tests cover every ordering among claim, host guard, record projection, trash marker, prefix deletion and resumed release. Existing completed tombstones remain the app lifecycle input, not authority to bypass an occupied claim. These objects also survive app-prefix purge. |
| + |
| +Writer inventory at dcf0cad: provision.CreateRecord, UpdateRecord, SetRecordRegion, SetRecordMove and DeleteRecord; console/apiinstances.go (create/shape/delete), apiinstancemove.go:136 (console move), appsettings.go:725-740 (password gate), channelmove.go, fleets.go, domainattach.go and apidomains.go (detach/delete); host/movespass.go; console/ownership.go and release/trash.go. PurgeApp's raw List/Delete sweep at trash.go:228-241 bypasses DeleteRecord today and MUST be restructured to conditionally delete instance records through the guarded primitive; its unattended caller domains/reconcile.go:232 must share the lifecycle exclusion below. A guard in the interactive console door alone is insufficient. Console/server.go's canary retirement cannot remove an active operation's required channel/artifact availability. |
| + |
| +Record parsers/listing consumers are a separate audit population: provision GetRecordETag/ListRecords/ListAllRecords/Reconcile; console/relayops.go:795, apiinstancemove.go:43, apiinstances.go:384/:467/:824/:1025 and fleets.go:624 (including its raw prefix list); edge/storedoor.go:477's instances/*.json write refusal. Registry-projection/admission consumers include host/steering_loop.go and agent.go, edge/edge.go and alias resolution, activator begin/serve/BeginSidecarRequest/ServeScheduled and sweepOnce (both scheduled wakes and awake sidecar restart), host restart/restartscope, config SyncExec/fingerprints/materialization, release/root adoption and system rollback. They do not all parse Record JSON. Reconcile's retirement of registry rows after record disappearance/trash must preserve the independently reconstructed hostname denial authority. |
| + |
| +The implementation inventory must also search raw prefix deletes, JSON decoders and route materialization, not only named helpers. Tests cover both named record calls and low-level consumer fixtures; new host guards and lifecycle objects cannot be made reachable through edge/storedoor's app store namespace. |
| + |
| +To preserve normal continuous app deployments after maintenance, add a signed release-manifest declaration of supported runtime contracts and a bounded monotonic required-contract set in the outside-app guard. First resume specifies the required app contract (for example, a Home operator-schema contract or a Docs TeamMode authorization contract). The approved resume artifact must declare it; future normal adoption/wake may change artifact versions only if the verified signed manifest declares every required contract. Known older releases without the declaration refuse before starting. Ordinary record/config changes cannot remove a required contract; changing/removing a floor is a new explicit maintenance operation with its own reviewed compatibility checker and owner approval. Limit the set to16 canonical names; overflow refuses instead of dropping old requirements. |
| + |
| +This declaration is trusted release-author intent, not proof of code behavior. Real consumer compatibility and authorization tests plus independent review remain required for each claimed contract. It enforces rejection of known incompatible rollback artifacts while letting newly reviewed compatible releases follow the usual deployment path. It does not protect against a malicious trusted release author falsely declaring compatibility, which ordinary artifact signing also cannot do. |
| + |
| +The guard remains present after resume. Its resumed state binds the successful request/attempt, approved initial artifact/config, required contracts and last record identity. Service.ResumeFenced starts the exact approved initial artifact and replication under the existing lease with all admission closed. After its health/version checks, the worker CASes only that guard generation to resumed and updates the matching record projection. If either write fails, local admission remains closed and the started process is stopped through the checked path. If the guard write succeeded before the record projection and the agent crashes, a future wake requires both objects to agree; it does not infer completion. Recovery may finish that specific recorded resume only after rechecking exact pins/health under a lease, or remain fenced; it must never rerun the maintenance command. |
| + |
| +Only once both durable references agree may local admission open. Concurrent normal adoption continues to honor the active guard until then; after resumed it checks the required-contract set before a new release can replace the already-verified initial artifact. A process death after completed resume is an ordinary wake of a compatible signed release under the normal exclusive lease, not permission to return to an old incompatible artifact. A startup failure before completed resume stays fenced and has no successful-resume receipt. The exact CAS boundaries, crashes between them, channel promotion during startup, stale record objects and incompatible old manifests are required deterministic tests and deployed rehearsal cases. |
| + |
| +## Implementation and release gates |
| + |
| +1. Resolve the explicit seams above in a final reviewed design: complete record/admission consumer inventory; checked stop/replica-close evidence; lease-retaining checkpoint; cgroup/parent-death proof; environment binding; exact resume handoff. These are design work, not instructions to assume the current API already supplies them. |
| +2. Implement inert wire/state/privacy/owner-approval primitives with exhaustive CAS, replay, expiry, owner-removal and secret-canary tests. No runtime worker or public activation yet. |
| +3. Implement exec fence/checkpoint/runner behind an explicit disabled capability; deterministic race/crash tests at every durable transition, stale ready Route, schedule, sidecar, restart, adoption/config/move/delete, two workers, lease loss, missing DB, changed inode/config/release, command timeout/descendants and result overflow. Failures stay fenced and never seed. Meaningful mutations must defeat each authority boundary. |
| +4. Integrate CLI/API/review page and durable receipts, using isolated test credentials. CLI documentation is required by the repository gate; copy review remains waived by the user, other writing requirements remain applicable. Full native/cross-platform gate and actual exact-SHA Amadan CI, independent Fable whole-branch review and current hub merge workflow before rollout. |
| +5. Deploy a disabled capability to an isolated supported platform canary; verify actual agent/edge/console versions, then rehearse synthetic existing databases with an actual pinned consumer and fresh test-owner flow. Demonstrate no second writer, durable fail-closed interruption, private transport and checked resume. Only after deployed canary review may the capability be enabled for eligible real instances. No existing production DB maintenance is included in merely shipping this feature. |
| + |
| +The first rehearsal can use Home's synthetic operator receipt consumer and Docs' compatible sealing command. It is not a restore drill, real team enrollment, content sealing activation or suite-readiness claim. Those remain separate acceptance tasks using this supported boundary. |