amadan
Draft exec instance maintenance boundary and review prerequisites
1 file changed,
+65
−0
implementation/maintenance-exec.md+65 −0
CI cancelled — superseded by a newer push — run details
diff --git a/implementation/maintenance-exec.md b/implementation/maintenance-exec.md| new file mode 100644 |
| index 000000000..c8ec3ba1b |
| --- /dev/null |
| +++ b/implementation/maintenance-exec.md |
| @@ -0,0 +1,65 @@ |
| +# Fenced maintenance for an existing exec instance |
| + |
| +Status: draft for independent Claude Fable review. Platform discussion117/task14; base dcf0cad333405aa8990af62c0871bd6ef63d96ba. This is an implementation proposal, not authorization to run a maintenance operation. User authorization covers implementing the missing supported platform command as part of Oficina readiness. Concrete deployment, fresh owner approval and operation receipts remain required. |
| + |
| +## Source and scope |
| + |
| +Read the authoritative infra snapshot259913017c2aeb99554e2c1384c2117d279f6e2a: instance provisioning2026-08-04, exec config delivery2026-08-02, route-scoped config2026-08-16, exec accounting2026-08-07, SQLite corruption defence2026-08-15 and scheduled work2026-08-23. The current platform instance-move spec2026-08-28 supplies related lifecycle ordering, not permission to reuse move/cut: that operation later purges files and changes residence. Actual source at dcf0cad is authoritative for implementation seams. |
| + |
| +The first supported class is a provisioned, single-residence, hibernating, activator-managed exec instance with one canonical primary host and an existing local database. Refuse fleet placement, pools, unit/addr/static/service/backing/tunnel routes, non-hibernating instances, unknown shape, non-owned registry rows and an in-flight move. Aliases resolve to an explicitly displayed primary before approval; they never widen the operation to another database. No cross-box execution takeover, missing-DB restore, new DB creation, live seed, raw SQL, host shell or root command in this slice. Other shapes return unsupported without changing state. Prove current Home roles meet these checks through the new supported preflight; restart's asleep report is insufficient. |
| + |
| +The operation uses a shipped, verified release of the selected app, pinned by full version and artifact digest. It executes that artifact directly with a bounded argv array, never a shell, live symlink or caller-supplied executable path. The platform supplies the exact existing DB through its defined private command contract; the app owns schema/semantic validation, transactions and idempotent operation receipts. This is existing instance authority under the platform's accepted shared-uid exec isolation model, not a claim to sandbox a hostile app binary from sibling files. It grants no root or deployment credentials. Home's owner receipt package and Docs docs-seal remain separate reviewed consumers. |
| + |
| +## Owner authorization and command interface |
| + |
| +Add `carlos maintenance prepare`, `status`, `result`, `retry` and `resume` under one documented command. All account/app/primary-host selectors are mandatory and exact; no default wildcard or account-wide variant. Prepare validates a JSON input file (0600, regular file, bounded), uploads it over the authenticated console API, and returns an immutable request ID plus a console review URL. CLI credentials are isolated in tests through carlosTestEnv and scratch cwd. Public status/result metadata never echoes argv, private input/output, secrets or app stderr. |
| + |
| +An ordinary CLI principal has no authentication-time proof (svcprincipal.Principal). Do not treat a recently minted bearer, service minter address or app-token instanceGate as fresh owner authority. Only a current human owner can prepare/read these operations. The console approval page shows exact primary host/account/app, operation label, artifact version/digest, configuration binding and downtime consequence. Its mutation gates run session -> CSRF -> current owner byte-identical404 -> existing requireFreshAuth last. Approval binds the immutable request digest, selected instance record revision and preflight revision; a changed request requires a fresh approval. The existing Keymail step-up supplies the actual gesture. No agent fabricates it. |
| + |
| +Prepare is inert until this browser approval. Approval is one-time and expires before a queued request can start indefinitely: five minutes from the actual owner gesture, checked before accepting the fence request. Once a matching accepted fence exists, recovery may continue stopping/checkpointing safely without another gesture; starting a new command attempt, changed inputs, retry and resume each require their own current-owner fresh approval. A lost CLI connection does not cancel, rerun or resume anything. Status polls immutable receipts; command attempts have separate IDs and never arise from GET or polling. |
| + |
| +The public request record holds random request/attempt IDs, binding digests, bounded operation label, timestamps and state only. Encrypt argv and private input immediately using the existing deployment secret infrastructure with a NEW purpose-separated authenticated envelope bound to account/app/host/request/attempt/direction and envelope version. Do not reuse an existing config envelope with different semantics. Public binding hashes cover randomized ciphertext, not low-entropy plaintext. Reject absent/decrypt-failed keys; do not log decrypted fields. Input is capped at64KiB, argv at32 entries/8KiB total, execution at five minutes by default with a hard ten-minute ceiling. Private result is capped at64KiB. App output on stdout/stderr is discarded (not teeStd); a dedicated private result FD carries the bounded response and is encrypted before durable publication. Result retrieval requires current human-owner authority and writes only to an explicit exclusive0600 local output file, never stdout. No retrieval capability is exposed in a public receipt. Expired private payloads are deleted after24h; nonsecret receipts and ciphertext hashes remain for audit. A result lost before durable publication is ambiguous, not a success with an invented empty response. |
| + |
| +## Durable state and all consumers |
| + |
| +Use an additive optional maintenance object on provision.Record, mutated through narrow compare-and-swap helpers. It points to immutable request/attempt envelopes and records a monotonically increasing fence generation, primary host, owning box identity, expected record/config/artifact binding and current phase. Permanent operation receipts live in a separate account/app/host namespace; completing one does not erase its generation or history. No unbounded list is embedded in the instance record. Old records with no maintenance object retain byte-identical behavior. |
| + |
| +Phases are awaiting-approval (console-private intent, not yet a route fence), requested, quiescing, fenced, executing, succeeded, failed, ambiguous, resume-requested and resumed. requested through resume-requested all deny serving. A successful command stays fenced. Terminal failure never auto-clears. `executing` is durably committed before spawning; after restart an executing attempt becomes ambiguous and is NEVER automatically repeated. CAS binds phase+attempt+generation+owner; stale workers cannot overwrite new attempts or clear a later fence. Control serial nudges convergence, never supplies authority. |
| + |
| +Inventory every parser and writer of Record, not only the new handler: provision Get/List/Create/update/delete/shape/channel/environment/region/move, console account/app deletion and restore, root adoption, edge registry reconstruction, restart workers, config sync, scheduling and retention. All mutation writers preserve the additive object and reject conflicting move/delete/shape/environment/channel actions while fenced unless they are the separately authorized resume transition. App-wide release promotion may proceed for siblings but cannot change this instance's pinned executable or environment. No config sync/removal/reap or adoption symlink change may affect a fenced instance. The implementation must pin each audited consumer in tests; an incomplete inventory blocks implementation approval. |
| + |
| +At boot, reconstruct the local durable fence from authoritative records BEFORE enabling HTTP, schedule or sidecar wake. Unknown phase, failed record read, missing active receipt or unsupported maintenance version fails closed for that instance. Enforce admission inside activator begin/wake and sidecar/scheduled paths, not just edge Status. A caller may hold a stale ready Route; after acquiring the normal per-host lease it must re-read/check maintenance authority before starting the child. This closes absence-read -> maintenance-request -> lease-acquire races. Edge returns fixed503/no-store for a fenced route, including aliases; timers remain recorded but cannot wake it. Unknown/missing instance authority never becomes a fresh seed fallback. |
| + |
| +Only upgraded maintenance-aware agents may serve an instance once maintenance is enabled for it. Deployment preflight verifies the capability and version of every eligible residence/serving box, the active primary and the platform rollback floor. An old platform binary that ignores the new record is NOT a safe rollback while a fence exists. This is an explicit operational release floor, not a property an older parser can enforce. Failover can observe the fence but cannot execute on a different box or recreate a missing local DB. Loss of the pinned box leaves the operation fenced and unavailable until a separately reviewed recovery path exists. |
| + |
| +## Exclusive execution boundary |
| + |
| +The worker first verifies its current ownership of the exact provisioned primary route, supported class and request/config/artifact binding. Region equality and Quiesced alone are insufficient: Quiesced only proves the local activator map has no entry. It writes the durable fence, marks local admission closed and retires the child/sidecar. It must obtain explicit success evidence that both are stopped and the old replication store is closed; current stop logs and deletes the entry even after Stop/Close errors, so ordinary RetireExcept+Quiesced cannot be treated as that proof. Add a checked stop result rather than weakening the existing routine stop semantics. |
| + |
| +Acquire the existing per-host replica lease only after the above closure, and retain it continuously through pre-checkpoint, command execution and post-checkpoint. A lease held elsewhere means wait/refuse, never force-take. Add a narrow lease checkpoint primitive that opens/syncs/closes a temporary replication store while the SAME Held renewer continues to own the lease; it neither releases nor creates a second concurrent replica. Reuse the existing replica-store construction and SyncAndWait implementation. Do not use Replicator.Close (which releases leadership) between these phases. The primitive checks ownership before/after, surfaces close/sync failure and cannot operate on a spent Held. Its deterministic tests prove no acquisition gap and one replica at a time. |
| + |
| +Under that lease, validate the exact platform-derived existing DB path: regular file, no symlink substitution, within the selected DataDir namespace, expected device/inode/file identity from the stopped preflight, expected route DB path and no quarantine/corruption state. Refuse missing/changed files without Restore, Seed, create, unlink or replacement. A checked pre-command checkpoint proves the retained replica carries the stopped tail. Record a nonsecret DB identity/checkpoint receipt, not document contents. The app receives only the existing DB path; its command must itself refuse creation and validate any key fingerprint/semantic preconditions before mutation. |
| + |
| +Run the pinned artifact under the existing selected instance cgroup/tier and approved merged environment snapshot, stripped of platform-only/control/purge variables exactly as the established exec boundary requires. Do not call EnsureAdminToken or silently create config during maintenance. Resolve and verify the artifact before execution without depending on the mutable live link. Capture an immutable encrypted environment binding during approved preflight, not a plaintext public hash; worker verifies it again before spawn. Concurrent environment/instance overlay changes fail the attempt instead of silently switching keys. |
| + |
| +Persist executing before spawn. Bound the whole process tree and private output; timeout, output overflow, lost lease, cancellation, parent death or unexpected descendant forces termination and an explicit failed/ambiguous receipt. Use cgroup-wide stop/emptiness proof plus normal Wait; PID alone is not death evidence. Monitor lease health with a deadline before lease expiry, not just eventual error polling. Parent/agent crash must not leave an orphan writer eligible to run after a new lease is acquired. Platform cgroup/parent-death behavior must be verified or extended before this path is allowed; unavailable proof leaves it fenced. No result exit0 can substitute for checked empty process tree. |
| + |
| +After a successful app exit, revalidate the expected DB identity policy (in-place SQLite VACUUM is allowed only if the reviewed operation's identity rule explicitly permits its file behavior), run checked integrity validation and a final checkpoint under the retained lease, then atomically publish encrypted private result plus succeeded receipt referencing its digest. Publication failure is ambiguous. Failure paths may preserve a partially mutated DB; never auto-restore, delete historical replicas or claim rollback. Release the lease only when command/replica shutdown is proven and the durable fence still denies every wake; otherwise keep/abandon leadership conservatively and record ambiguity. No command reruns automatically after crash. |
| + |
| +## Retry and resume |
| + |
| +Retry creates a new immutable attempt with a fresh owner approval and the previous attempt receipt bound into its request. The app must supply idempotent operation semantics or explicitly describe repair; platform cannot infer that a failed command made no changes. Before retry, re-prove stopped processes, current record generation, exact DB/artifact/config and checkpoint. The same request ID with changed bytes conflicts. Reading a finished receipt never executes anything. |
| + |
| +Resume is a separately approved operation, not `clear maintenance`. It names an expected serving release/artifact, configuration/key binding and successful app-specific compatibility-check command from that release. Platform verifies all pins under the fence/lease, runs the non-serving checker against the existing DB, verifies integrity/checkpoint and records its result. The app checker owns format/key/authorization compatibility; platform cannot infer it from exit of an earlier migration or a byte-compatible old binary. Docs TeamMode's minimum authorization-aware release floor is one such consumer requirement. |
| + |
| +Only after the checker succeeds may the worker publish resume-ready evidence, install the pinned serving release/config and arrange the lease-to-serving replication handoff. Clear the fence through one guarded transition bound to this evidence; old/new worker races cannot resurrect a superseded resume. Admit HTTP only after the exact new process and replication are healthy and report the approved version; retain a durable resumed receipt. If process startup/replication/version verification fails, return to fenced/failed, not ready. The implementation must decide and test the precise lease handoff before code approval; simply releasing and hoping the next wake uses the approved pins is insufficient. |
| + |
| +## Implementation and release gates |
| + |
| +1. Resolve the explicit seams above in a final reviewed design: complete record/admission consumer inventory; checked stop/replica-close evidence; lease-retaining checkpoint; cgroup/parent-death proof; environment binding; exact resume handoff. These are design work, not instructions to assume the current API already supplies them. |
| +2. Implement inert wire/state/privacy/owner-approval primitives with exhaustive CAS, replay, expiry, owner-removal and secret-canary tests. No runtime worker or public activation yet. |
| +3. Implement exec fence/checkpoint/runner behind an explicit disabled capability; deterministic race/crash tests at every durable transition, stale ready Route, schedule, sidecar, restart, adoption/config/move/delete, two workers, lease loss, missing DB, changed inode/config/release, command timeout/descendants and result overflow. Failures stay fenced and never seed. Meaningful mutations must defeat each authority boundary. |
| +4. Integrate CLI/API/review page and durable receipts, using isolated test credentials. CLI documentation is required by the repository gate; copy review remains waived by the user, other writing requirements remain applicable. Full native/cross-platform gate and actual exact-SHA Amadan CI, independent Fable whole-branch review and current hub merge workflow before rollout. |
| +5. Deploy a disabled capability to an isolated supported platform canary; verify actual agent/edge/console versions, then rehearse synthetic existing databases with an actual pinned consumer and fresh test-owner flow. Demonstrate no second writer, durable fail-closed interruption, private transport and checked resume. Only after deployed canary review may the capability be enabled for eligible real instances. No existing production DB maintenance is included in merely shipping this feature. |
| + |
| +The first rehearsal can use Home's synthetic operator receipt consumer and Docs' compatible sealing command. It is not a restore drill, real team enrollment, content sealing activation or suite-readiness claim. Those remain separate acceptance tasks using this supported boundary. |