carlos / platform Public
Always-awake exec lifecycle for Home
Status: design proposal for LLM implementation and independent review. No code or new authority is approved by this document. Revision 3 addresses Fable’s seven original requested changes and the two integration findings on 09b92e8. Independent rereview remains required before implementation; deployment additionally requires the explicit gates below. User copy review is waived. Platform task 32/discussion 117 and Home discussion 44 track this amendment.
Base: platform 11a303996d5503d29ede77598243ddc66f735279, which merges the Fable-approved inert maintenance components e87b410. Home requirement: 7e282e2a363b9cc90cd74dd97809e6613a699af8, docs/superpowers/specs/2026-08-30-home-design.md §2.1 and AGENTS.md; both are supplied with the review bundle. Maintenance design 6a81dac, implementation/maintenance-exec.md, is approved. Guard/claim/RecoverClaim, Held lease, strict containment, local fence and scheduled-token primitives now exist in this base. They are unmounted components: the maintenance worker, browser approval workflow, preflight, end-to-end resume (including proposed ResumeFenced), checker launch adapter and enforced platform floor are NOT implemented. References below to the maintenance workflow mean that approved/proposed integration contract, not an existing executable service. No component approval proves deployment capabilities.
Required behavior
Home stays running without incoming traffic. The platform starts an eligible Home process at boot, keeps it through idle periods, and proactively recovers it after an ordinary crash or adoption cycle. Maintenance, lost ownership, shutdown and an unproven cleanup still stop it. A failure can cause an explicit outage; “always awake” does not promise active-active availability or zero downtime. It means no deliberate idle sleep and no dependency on a visitor to recover service.
Preserve the current exec database and socket paths, app account/host identity, process user, config sources, replication class, lease key and replica prefix. Do not create another Home identity or directory, transfer keys, reseed an existing database, or move data into a new state directory as part of this change. Credential and directory roles remain separate primaries with separate state and configuration. An alias never owns a second process.
Supplied API receipts from 2026-09-08 show three effective Home routes with hibernate=true, unit=null, and status=ready. The instance records' absent hibernate fields resolve to the platform default; absence does not mean no hibernation. The observed primaries are home.bds.oncarlos.com, overnight-home-id.bds.oncarlos.com, and overnight-home.bds.oncarlos.com; these exact host/account/app identities are inputs to subsequent preflight, not authority to migrate them. These receipts identify a topology gap, not current process liveness or maintenance eligibility. Their local audit is /home/paulca/.cache/oficina-home-topology-audit/topology.json; deployment must obtain fresh supported preflight evidence.
Existing mechanisms and choice
The current exec manager has the right data ownership but the wrong residency behavior. edge.Edge.Reload selects management using Kind == instance && Hibernate; Service.serve repeats that condition. RetireExcept stops a child when that flag is cleared. sweepOnce stops idle children, and after a crash or artifact cycle leaves the next wake to a request or due timer. ConfigFromEnv requires a positive idle duration. None of those mechanisms supplies always-awake residency.
An ordinary unit service is already a supported way to avoid idle hibernation. Home's cmd/home/main.go explicitly describes that original deployment: serve, inherited listener, and STATE_DIRECTORY/home.db. The carlos-app@.service template has Restart=always and a boot target. Enabling the service at boot matters: enabling only its socket still permits demand startup, and a listening socket does not prove a healthy process. Adoption's try-restart also does not start a deliberately stopped service.
Choose an explicit always-awake activator exec policy for the current instances. Migrating them to units would change DynamicUser/state-directory/config ownership and replication management. UnitBackend.Acquire returns an inert lease; unitRep.Close is a no-op; root litestream owns replication. Its unitProc.Exited records that Stop was called, not kernel process-tree death. The socket can reactivate the service even when the agent is down. Consequently unit migration also needs a separately reviewed root/unit/socket maintenance protocol and data migration. Approved maintenance6a explicitly excludes that class. Extending the existing exec manager's residency policy preserves more of the currently relevant ownership and maintenance machinery.
Do not approximate residency with requests, timers, scheduled jobs, dummy sidecars, an excessive idle timeout, or a hibernate=false toggle. Do not describe those as satisfying Home §2.1.
Record shape and public policy
Add optional string lifecycle to provision.Record and the registry route projection. The only nonempty value in this slice is always-awake-exec-v1. Absent/empty means the existing behavior byte for byte. Unknown versions refuse before route materialization or startup; they never degrade to an unmanaged proxy.
A record carrying this value must also carry explicit hibernate:false. The two fields are written by one record CAS and materialized together in one registry transaction. Contradictory hibernate:true refuses. The new field describes a real managed lifecycle; the false flag accurately reports that the instance does not hibernate.
Required eligibility: an existing provisioned instance; derived canonical local Unix socket and nonempty canonical primary database path; ordinary exec backing; exactly one verified residence; ready status.
Refused shapes: any unit, TCP upstream, static/service kind, backing override, tunnel, fleet, pool, in-flight move, any capability requiring unit execution, or any granted port (grantedPortTokens nonempty). Resolve the actual capability set and grants in preflight; an unknown grant state refuses. Granted ports can start a socket-activated unit beside the exec child even without Daemon. The serving box must be the currently verified owner. Preserve existing account/app/host/creation identity and replication class. Unknown placement or owner refuses. Do not infer single residence from an empty region string without authoritative placement resolution.
This first slice is an opt-in for existing initialized instances, implemented through the fixed platform maintenance action below. Clean installation and new Home credential/directory roles need a separate supported always-awake provisioning path; that remains an outstanding suite requirement. Existing-only migration is not full Home readiness. It does not add a create-time always-awake option, an opt-out, a unit conversion, or a generic command runner. Once activated, clearing or changing lifecycle, flipping hibernate, changing its execution class, or starting a move is refused by ordinary writers. Retirement/trash/purge still use their reviewed lifecycle protocols. A later policy downgrade or relocation needs its own reviewed transition; “legacy” is not an escape hatch.
Propose one supported control operation:
POST /api/cli/apps/{account}/{app}/instances/{host}/lifecycle
{"request_id":"<canonical operation ID>","lifecycle":"always-awake-exec-v1"}
carlos instances lifecycle -app <app> -host <primary> always-awake-exec-v1The CLI generates and retains a cryptographically random 32-byte canonical base64url request ID before its first POST, using maintenanceguard.ValidID. Repeated preparation for that ID returns only the same authorized account/app/host/action request; a different binding refuses. The ID supplies idempotency, never authority. This is a proposal for review, not an existing endpoint. Resolve accounts with the existing CLI/account conventions and retain the family's indistinguishable not-found behavior. Require a current human account owner, using existing principal and ownership machinery; a service/app bearer cannot enable a new process lifecycle. Browser exposure, if added, uses the existing session → CSRF → current owner order. This does not require or substitute for maintenance's separate fresh owner approval. Do not broaden the generic PATCH whitelist or expose operator-only columns as an incidental change.
This endpoint prepares an inert fixed-action maintenance request; it does not immediately change policy. The CLI enters the maintenance design’s browser approval workflow (not yet implemented), then reports recorded outcome, and explicit resume remains a separate fresh approval. After authorization, preparation requires a supported, bounded platform preflight for this exact record revision and owner. The preflight proves current initialized DB identity and existing replication/config ownership, eligible platform capability/floor, and no active maintenance claim/guard or move. Do not obtain those facts by sending the caller to a box shell. Preflight cannot mint a database, lease, or token. Record CAS conflicts invalidate it. No arbitrary artifact/config/command arguments are accepted.
Reissuing the same request ID reports its existing immutable request/outcome after revalidation. A new request for an already installed policy reports that state and cannot acquire another operation or resume an unfinished transition. A lost CLI connection never creates another random operation ID. The response distinguishes “prepared”, “policy recorded but fenced”, and “healthy resident process observed after explicit resume.” A bounded CLI wait reports the actual status/version or timeout without undoing an accepted operation. A timeout is not permission to retry its effects. There is no claim of successful service startup merely because the CAS succeeded.
Every record writer preserves the lifecycle field as well as the maintenance projection. Generic record PATCH, placement, delayed channel changes, headless writers and deletion must use the maintenance writer inventory. Reject conflicting mode/identity changes before preceding SQL/registry/replica side effects. Active maintenance continues to block conflicting channel/environment changes. Ordinary approved adoption after resumed may still change compatible app releases.
Ownership and sleep are separate predicates
Define one shared pure validation result from a decoded route policy. Its concepts are:
ManagedByActivator: existing hibernating instance shape, or eligible explicit always-awake exec shape.MayIdleSleep: existing hibernating behavior only; false for the new policy.OwnsExecReplication: validated activator-managed exec backing AND nonempty canonicalDBPath, including the new policy; unit backing remains excluded. The path condition remains mandatory in bothops.FleetDBsWithSkippedandops.HibernatingHosts. A malformed new-mode route refuses; a legacy route with no database stays outside this predicate.RequiresResidentProcess: explicit always-awake exec policy only.
Concrete Go names may follow local conventions, but no consumer may reconstruct these with its own !Hibernate rule. Malformed/unsupported policy returns a refusal, not four false values. A legacy false-hibernate socket remains externally managed and must not suddenly acquire a child.
The registry update must preserve the replication rung. Existing Registry.SetHibernate(false) executes hibernate=0, replication=''; calling it during this transition is wrong. Add a dedicated checked transactional projection of lifecycle+hibernate with the existing replication value preserved, or extend the canonical projection transaction to do exactly that. Do not alter legacy SetHibernate behavior for unrelated routes. The invariant is Replication != "" implies OwnsExecReplication, rather than implies Hibernate. Apply this invariant to BOTH Registry.Upsert INSERT normalization and its ON CONFLICT replication expression, as well as the dedicated projection transaction. Update replicationRail and carlos add validation/defaulting to the same predicate; neither may accept a rung for an empty DBPath or externally managed route. Preserve the existing rung when an eligible resident route is re-upserted with no new rung. Reconcile’s existing-owned branch must use the dedicated projection instead of SetHibernate(false), including after record CAS succeeded but projection failed.
Before convergeOwned or ANY owned-row field update in Reconcile, compare the authoritative record lifecycle with the existing row. A row already carrying v1 paired with an absent, unknown or conflicting record lifecycle is an invalid regression, including a stripped legacy record. Quarantine it: preserve the entire v1 row and replication ownership, install a distinct local policy-regression admission/start denial keyed to the exact primary identity and observed record/row revisions, and cancel pending starts. This is a new denial reason, not a fabricated maintenance generation. Reconstruct it from the persisted row plus authoritative mismatch before admission after restart; normal maintenance resume cannot clear it without resolving the exact mismatch through its approved projection. Keep that primary in the activator-owned retention set so Edge.Reload/RetireExcept cannot reinterpret it as external ownership. Stop any running main/sidecar and replica through the checked stop path; retain uncertain resources and deny replacement on failure, with no premature lease release. Even after a proven stop, retain the row so root replication continues to exclude its DB. Record/row disagreement supplies neither ordinary startup nor a policy repair permission; repair requires the exact approved maintenance operation. This rule differs from temporary nonauthoritative discovery failure, which does not fabricate a regression or retirement.
ops.FleetDBsWithSkipped currently excludes exec DBs only when Hibernate is true. Replace that exclusion with OwnsExecReplication; otherwise the root replicator may pick up a database already streamed under the activator lease. The generated fleet replication config must be checked before activation, and missing/unreadable ownership must refuse a potentially duplicate replication change. This mode does not rename HibernatePrefix or move existing objects.
Proactive resident reconciliation
Add one bounded resident reconciliation loop to the existing agent/activator lifecycle. A proposed internal interface is Service.ReconcileResidents(ctx, desired) where desired is a complete locally-owned set produced from freshly validated authoritative records and registry projection. It is not a public wake permission and does not accept a caller's arbitrary Route as sufficient authority.
The loop reads SQLite and durable authority outside a.mu. Under the existing mutex it installs/cancels per-host desired entries identified by exact account/app/host/record identity, record revision, owner and local desired generation. An asynchronous result must compare that identity/generation again before becoming the current process. Removal, mode mismatch, changed owner or active guard cancels pending starts and retries. A stale complete snapshot cannot overwrite a newer one; authority failures are represented explicitly rather than fabricated as an empty successful listing.
Startup order is: establish agent containment and owner prerequisites; read authoritative policy/guards; reconstruct local denial; install validated desired entries; start eligible residents through normal lease acquisition; then report readiness for each host. HTTP and schedules may be mounted earlier only if the affected host remains denied until its policy/guard state is known. A cached pre-restart Route is never enough to spawn. Guard read failure or an unsupported version denies that host. Already-running ordinary service must not be stopped solely because a nonauthoritative listing temporarily failed; current lease-health and existing approved guard freshness rules continue to apply. The implementation must distinguish incomplete discovery from authoritative retirement.
Coalesce proactive starts, incoming requests and due-work starts through the same per-host waking entry. An incoming request cannot bypass resident backoff or an uncertainty latch. Keep one startup attempt per host and a global ceiling of four proactive starts. Poll due residents every two seconds; use per-host backoff of 1, 2, 4, 8, 16, 32, then 60 seconds with bounded 0–20% positive jitter. Use injectable clock/jitter for tests. An accepted successful healthy run lasting 60 seconds resets the failure count. Request arrival and serial bumps do not reset it. These are proposed reviewable defaults, not a promised recovery SLO.
Retain the existing five-minute wake-attempt ceiling and thirty-second request wait; use the existing bounded stop/replica closure paths. The loop never holds a.mu across registry, object-store, filesystem, process Wait or health I/O. Cancellation must reach actual operations; no unbounded goroutines around uncancellable calls. An admission timeout is refusal, not proof that the old process stopped. Any retained cleanup uncertainty blocks replacement until the exact prior resources are proven stopped through the supported recovery path.
The normal exec sequence remains exclusive lease → current desired/guard recheck → validated existing DB/replica → start → replicate/health → publish awake entry. For this mode, missing local DB may use the existing validated replica restore under the lease. If the replica is also absent or cannot be validated, refuse. Never call Seed or initialize a fresh Home database as resident crash recovery. Existing first-boot provisioning remains separate and completes before this opt-in.
Shutdown first closes service-wide startup admission and cancels the resident loop and every in-flight wake under the same synchronization used to publish entries. Join each exact wake operation, including one already inside backend Start. If Start returns a resource after cancellation/shutdown, transfer it to checked cleanup rather than publish awake: prove main and sidecar death, checkpoint/close the replica, then close the held lease. Join cleanup before declaring successful shutdown. The current actWaking “acceptable orphan” branch is explicitly forbidden for this mode. Deadline expiry reports unproven shutdown and retains uncertainty; it is not successful cleanup or permission to release ownership/start a replacement. Deployed supervisor containment remains a separate gate when a process exits before cleanup completes.
The idle branch consults MayIdleSleep; lease loss, process death, route removal, owner change and agent shutdown still stop. After a successful ordinary stop for app death, artifact adoption or explicit ordinary restart, an unchanged eligible resident desired entry becomes retryable without waiting for traffic. A new version continues to respect existing adoption/health semantics and the current ninety-second convergence grace/rate limit; a resident restart joins the same four-start ceiling. A stop failure is not a reason to race a replacement.
Lease loss stops immediately through the existing checked path. A later normal retry revalidates owner and acquires a fresh lease; it cannot extend or resurrect the lost one. During maintenance the retained lease belongs to the maintenance protocol, and ordinary reconciliation cannot acquire it as a substitute for explicit resume.
Maintenance amendment
Replace maintenance6a's first supported-class restriction with this exact meaning after review:
> A provisioned, single-residence, activator-managed exec instance with one canonical primary host and an existing local database is eligible only when its validated lifecycle is either the existing hibernating on-demand exec lifecycle or always-awake-exec-v1. Unit, address, static, service, backing, tunnel, fleet, pool, unknown lifecycle and in-flight move shapes remain unsupported. An unflagged legacy non-hibernating route is not an always-awake managed instance.
All owner approval, request/record/config/artifact binding, app lifecycle claims, lease deadlines, checked stop, cgroup emptiness, private payload, restore verification, execution receipts and explicit resume requirements remain unchanged. The eligible-class extension alone adds residency and grants no new maintenance command or operator power. The fixed-action section below separately proposes new action and recovery authority and requires its own explicit review.
The proactive loop must participate in the same admission mutex/fence protocol. Installing a fence cancels queued and in-flight resident startup and blocks future retries, independent of a requires resident desired bit. A start already returning a resource handle remains owned and must be cleaned through the checked path. All startup side effects, including environment/token preparation and sidecars, must occur on the allowed side of the appropriate admission/guard boundaries. The separate scheduled-token fix covers only its local ordering; it is not proof of complete materialization protection.
An executing, failed, ambiguous or resume-requested maintenance record never becomes an ordinary crash to retry. Persisted guard reconstruction wins over desired residency after an agent restart. Recovery may finish the exact supported cleanup/receipt transition; it must not replay a command. Only the reviewed resume protocol once implemented, with matching guard+record projections, retained lease and verified health, opens admission. After that transition the always-awake controller maintains the already-authorized healthy process.
Fixed platform residency-transition action
This section is a NEW proposed maintenance action amendment. It is not an app-exec operation and is not already authorized by maintenance6a. It makes activation concrete without adding another maintenanceguard.OperationKind: the existing MaintenanceOperation app claim provides exclusion, while a separately versioned typed request selects this fixed platform action. Fable must approve both its authority boundary and its implementation before it is exposed.
Add a closed request discriminator for platform-residency-transition-v1, distinct from app-command requests. Reject unknown action/version and every command, argv, executable path, raw config, environment override or caller-supplied record patch in this variant. Its inputs are exact account/app/primary host, immutable request ID and the platform preflight reference. The platform derives the only allowed application-field delta from the approved old record: lifecycle="always-awake-exec-v1", hibernate=false. Identity, paths, region, replication class, channel, environment, credentials and all other record data remain unchanged. Maintenance projection updates are additional protocol metadata, not part of the two-field policy delta.
The platform verifies its own runtime capability/floor for this action. An app manifest's advertised maintenance command does not authorize it, and it does not dispatch through the app-command executable path. The approved existing app artifact/config remain pinned for restart. Existing required app semantic contracts are preserved unchanged; this action cannot lower any guard or binary floor.
Owner approval follows maintenance6a: current owner/non-disclosure and CSRF gates before a fresh authentication gesture, five-minute acceptance window, and approval bound to the immutable request/preflight, exact old record revision/identity, old execution policy, derived two-field delta, current owner box, pinned app artifact/config and required capability evidence. The UI names the fixed change and its downtime. No approval is inferred from preparing the request, the lifecycle CLI, an existing bearer, or a GET.
The worker performs the following ordered protocol:
- Recover or acquire the exact nonexpiring maintenance-kind app claim using the request's operation ID. A competing claim or lost CAS has no side effects. Recheck an active untrashed app, original eligible hibernating exec identity/revision, owner, no move and current authenticated floors. Existing resumed guard history is advanced through its normal CAS; never replace it or drop required contracts.
- Create/advance the host guard to requested with the typed action reference; CAS-install the matching maintenance projection on the exact record. If the projection fails, leave guard and claim held and report failure. The old application-field snapshot and derived target are immutable private request material. The worker tracks actual record revisions produced by projection updates, rather than treating the pre-guard ETag as still current.
- Quiesce through the existing local fence. First prove original app and sidecar shutdown, old replica closure and kernel group emptiness through the reviewed maintenance protocol. The old normal replica owns its lease release; do not acquire the maintenance Held lease before that closure is proven. Only after those proofs acquire the exclusive maintenance Held lease, recheck authoritative owner/guard/record and retain it for the transition. An old lease still held is a refusal to proceed, not permission for parallel ownership. No proactive resident loop, root adoption, config sync, scheduler, socket fallback or ordinary request may start a process. Recheck exact guard/claim/record and app state under that authority. In the source class, the record still has its old hibernating policy.
- Persist executing before changing policy. This is a platform record transition with no app subprocess, shell, SQL migration, seed or restore. The existing checked checkpoint/integrity requirements still protect the stopped database. The typed handler performs one CAS from the current exact guarded record to the derived target fields while preserving its current maintenance projection and every other field. A conflict does not retry against a newly read arbitrary record.
- Read back the actual resulting record revision and compare all protected fields to the approved snapshot. Publish an immutable action receipt naming original identity, approved request/delta, pre/post record revisions, target policy, current claim/guard generation and the checked stop/lease evidence references. Sensitive snapshot material remains in the existing private envelope; public status contains no environment or DB contents. Receipt publication failure is ambiguous and leaves the fence held.
- Materialize the target lifecycle and false Hibernate in one checked registry transaction while still fenced, preserving DB/socket/owner/replication columns. It must be the projection of the exact target record revision, not a freestanding local edit. Read back and verify it. No ordinary manager adoption can open admission. A stale projection is a repairable incomplete action only under the exact operation recovery below.
- After matching target record/registry state and the normal final checkpoint/receipt requirements, publish succeeded with immutable action evidence. Successful policy installation remains fenced. Report that state; do not release the app claim or automatically start Home.
- Explicit fresh owner-approved resume validates the target policy under the amended eligible class and the exact target record, selected serving release/config, capability floors and action receipt. Before serving, run the fresh app-specific non-serving compatibility checker from that selected release under the fence and exclusive lease, exactly as maintenance6a requires. Bind its successful receipt to this resume attempt, selected artifact/config and current database/record identity; complete its normal Wait and checked process-tree cleanup. The policy action itself still invokes no app subprocess, but resume retains this separate mandatory checker. Only then start the approved serving app under the retained exclusive lease and checked resume protocol with admission closed. The accepted ResumeRequested guard MUST carry a nonempty sorted unique lowercase RequiredContracts set, derived from the exact selected Home release’s verified signed manifest and preserving all already-required contracts. The same set is bound to fresh owner approval and remains pinned through Resumed. At least one selected-release contract is mandatory; no available verified contract means resume refuses before any start. This action invents no unsigned contract or capability name. Verify health/version, publish the matching resumed guard and record projection, verify both, then open local admission and release the matching app claim only after serving health agrees. Install resident desired state as part of that handoff; it cannot precede or replace it.
No generic lifecycle claim enum is needed. The original maintenance claim stays held across requested, policy execution and explicit resume. This serializes trash/restore/purge and other maintenance for the app. Ordinary conflicting record/config/adoption writers still need the existing guard/record-CAS protections; the app claim does not magically serialize those writers. Platform transition code is a narrow additional authorized writer of the two fields, recognized only by this typed accepted operation.
The live registry transition now happens only with the old process stopped and all startup paths fenced. There is no online “keep the old child while changing ownership” half-transition. A policy-aware root replication generator excludes both the old hibernating shape and the new always-awake shape. Its floor must already be enforced before either persistent field changes. A separately running old generator is a preflight refusal, not an accepted race window.
Recovery of the exact transition
The table below applies only while the SAME live executor retains its valid exclusive maintenance lease and execution reservation. Rows involving policy CAS, receipt publication, registry repair or resume are unreachable without those retained resources. The same-executor owner-approval wait consumes the existing five-to-ten-minute bounded maintenance execution deadline; retries do not reset it. On expiry, those rows become unavailable, cleanup/uncertainty follows the parent protocol, and the host stays fenced. The renewer’s ability to renew indefinitely does not extend this operation deadline. It does not authorize post-crash recovery. If the executor dies or loses either resource, the first implementation has no supported recovery: the operation stays fenced and service remains unavailable. This is an explicit first-slice availability limitation, not an implied fresh-lease path. A separately reviewed executor handoff is required to remove it.
Claims never expire. RecoverClaim observes only the exact operation ID; status/GET is read-only. A new random request cannot take over an interrupted operation. Stopping/checkpointing cleanup under an accepted guard follows maintenance6a's existing limited recovery rule; it does not automatically apply the policy CAS or start an app.
Persist immutable expected old and target application-field snapshots excluding only the maintenance projection metadata managed by the protocol. Compare every other record field, including unknown preserved fields, against one of those snapshots on recovery. Do not infer success from hibernate=false alone. Record ETags must still be compared for each subsequent write. The implementation must use one strict canonical record representation for these protected comparisons and refuse unsupported/ambiguous encodings, not drop fields during normalization.
| Observed recovery state | Permitted action (same executor and retained resources only) | | --- | --- | | Claim accepted, guard/projection incomplete | Finish only safe fencing/cleanup for that exact operation; no policy application or startup from polling | | Executing/ambiguous, exact old application fields | No policy side effect is proven. A new fresh owner-approved retry creates a new attempt under the same held operation and revalidates current revisions; it may perform the one derived CAS. Never automatic retry | | Executing/ambiguous, exact approved target fields, receipt absent | The policy effect is present, but success is not proven. Explicit authenticated recovery rechecks exact claim/guard/identity and checked stopped ownership, records a recovered-observation receipt, and may repair only the derived registry projection. It cannot claim the old attempt exited successfully or replay the policy mutation | | Target fields plus valid action receipt, registry old/missing | Explicit recovery may re-project that exact record under the fence and record actual result; no app start | | Target fields and registry match, succeeded or recovered evidence | Fresh owner-approved resume may proceed after all resume preconditions; no new policy execution | | Any other record/config/artifact/owner or unsupported floor | Refuse, retain claim/fence, report conflict. No overwrite, reversal to hibernation, restore, seed or unit migration | | Guard resumed but record projection mismatches | Keep local and proactive admission closed; repair only the exact approved resume with fresh authoritative lease/health proof, as maintenance6a requires |
For this action, the proposed existing maintenance resume flow may offer recovery only after its read-only preflight classifies the exact approved target. Fresh current-owner approval binds that observed record/guard revision, recovery receipt kind, projection repair and intended resume. The worker repeats those checks before any repair. A change in the observation requires a new approval; status polling alone remains inert.
For this new action, explicit recovered observation of the exact target is a distinct receipt outcome, not a fabricated success exit. The proposed typed resume validator accepts succeeded action evidence or this recovered-target evidence after fresh owner approval and complete stopped/lease checks. Generic app-exec resume validation is unchanged. A matching target alone cannot satisfy it. Recovered-target evidence never replaces the fresh selected-release compatibility checker required before serving.
A process crash can lose the retained lease. Fresh-lease recovery requires TWO separate stopped proofs: the original app/sidecar/replica tree, and the previous maintenance executor. An empty Home cgroup does not prove that a paused platform worker cannot later publish a receipt, repair a record or launch resume. RecoverClaim observes the operation; it supplies no exclusive executor ownership. Lease expiry also supplies no executor-death proof.
Before any fresh takeover, the implementation must hold exclusive operation execution ownership and prove the previous executor quiescent or dead. Within one agent, this requires cancellation and completion/join of the exact worker while retaining a per-operation exclusive execution reservation; a cancellation request alone is insufficient. Across agent incarnations, it requires checked death/quiescence of the previous owning agent/executor and all command descendants, plus an exclusive operation reservation for the new incarnation. Reused PIDs, a new request/attempt ID, an unchanged claimed operation, the old Home process's death, elapsed time or a successful DB lease acquisition cannot substitute. Every later publish/repair/launch still rechecks the exact operation/attempt/generation and held lease; stale callbacks cannot outlive their execution reservation.
No existing primitive in the audited guard, Held lease or activator candidate proves this complete executor handoff. Therefore recovery that needs a fresh maintenance lease is explicitly UNAVAILABLE in the first implementation until this precise exclusive-executor and prior-executor-death proof is implemented and independently reviewed. A fresh owner approval cannot override that absence. Same-executor recovery while its valid lease and exclusive execution ownership remain held can follow the table; losing either remains fenced. The design does not invent an expiring claim, a new generic operation enum, or an arbitrary force-unlock.
If a later reviewed executor handoff provides those proofs, recovery stays on the pinned owner box and exact operation, under a separately fresh owner-approved resume/recovery attempt. It must recheck guard, identity, owner, stopped app/replica and then acquire a fresh exclusive maintenance lease. This is proposed future authority, not a permission to take over on another box. Counterexample: pause worker A after its last lease check with Home stopped; worker B recovers the same claim and acquires a new DB lease; resuming A would permit a late publish/repair/launch. B MUST refuse without A's checked quiescence/death and exclusive executor transfer, even when all Home-tree checks pass. The actual test must hold A at that boundary and prove no B-side mutation or start.
The remaining recovery implementation is bounded by existing maintenance lease deadlines and operation timeouts. It never attempts to make availability by bypassing an uncertain phase. This rule supplies a concrete safe refusal for a lost/failed box; restoring a replacement box and transferring ownership remains a separately reviewed disaster-recovery operation.
Activation and rollback
Activation is blocked until every record writer, serving box, edge, agent, root replicator/adopter and relevant recovery binary understands this policy and enforces its supported rollback floor. Additive JSON alone is not safe: an old writer strips lifecycle, an old edge treats false-hibernate as external ownership, and an old replication scanner can add a second replicator. An old route reader may also preserve stale hibernation state during partial conversion.
The signed app runtime-contract design is a separate pending dependency and does not itself define a platform binary floor. A distinct authenticated platform-floor design is still required. Independently, the reviewed signed app runtime-contract mechanism gates step 8 and EVERY subsequent wake: CheckServing must receive the exact verified manifest contracts and find the pinned RequiredContracts set. An unsigned self-report, only a platform floor, or an empty set cannot satisfy the Resumed guard. This design requires a distinct platform capability identifier for always-awake exec support in that reviewed mechanism; do not repurpose an app's maintenance semantic contract as a platform binary capability or trust a self-reported unsigned capability. Floors must precede both policy-writing activation and host/replication activation, even if no maintenance operation is enabled. Until a reviewed authenticated floor and its deployment proof exist, this mode remains disabled. This proposal does not define a new signing format or mark that dependency approved.
Deploy readers and deny/replication protections before exposing the policy operation. Activate a synthetic existing canary only after a wrong-version/writer negative control proves refusal. The fixed platform transition above requires checked downtime. Policy CAS and registry projection remain separate objects: during disagreement, serving and proactive startup deny. Repair only under the exact held operation, never by clearing lifecycle or starting another child. This is a reviewed-action proposal with a concrete ordering; it is not approved until the fixed-action discriminator, recovered-target receipt and fresh recovery lease rule have been independently reviewed.
After activation, old binaries cannot be an availability rollback. Roll back only to a verified policy-aware version that retains the same mode and data ownership; otherwise externally close traffic and follow a reviewed recovery procedure. Do not erase the policy, floor or maintenance history to make an old binary start. No command in this slice changes the host's DB path or transfers ownership to a unit.
Capability dispatch and checker launch
Carry ManagedByActivator and RequiresResidentProcess as explicit validated fields in caps.Plan from capabilityPlan and every other plan constructor. Before filesystem or unit mutations, refuse Set.Daemon && ManagedByActivator; checking only Hibernate would permit the false-hibernate resident to install a second service. For ManagedByActivator && RequiresResidentProcess, also refuse any nonempty grantedPortTokens before any write or unit action. Converge currently has NO Hibernate-keyed bypass for granted sockets; this refusal is new. Preserve the existing exec dispatch bypasses at adopt.go:772, adopt.go:1207 and agent.go:2640 for all activator-managed routes, and carry the new refusals through rollback snapshots and all plan constructors. Unsupported combinations refuse before capability writes. A signed manifest cannot override this execution-class restriction.
The selected-release compatibility checker uses the proposed checked maintenance launch adapter, with command/context and executable verification, not the ordinary serving path. Child fd4 must be explicitly closed and checked before Home’s database descriptor baseline; worker-held executable proof remains retained. The adapter and corresponding Home open-contract amendment remain unimplemented review dependencies. This proposal adds neither a path-based fallback nor approval for an absent checker.
Consumer inventory and tests
always-awake-consumers.json records exact base source functions currently branching on Hibernate, plus the serializer/control boundaries that must carry the new shape. It is an implementation checklist; a passing search count does not establish coverage. Every listed family needs a regression or an explicit unchanged-behavior assertion. The earlier maintenance writer audit supplies record deletion, lifecycle claim, config and prefix-purge ordering; this amendment does not replace it.
| Family | Required result | | --- | --- | | Record defaults/ValidateShape/Create/Update/update retry/Get/List; raw JSON consumers | Preserve/validate lifecycle and maintenance projection; unknown value refuses; no stripping on legacy roundtrip | | Registry schema/scan/Upsert/SetHibernate and provisioning mint/adopt/converge | Transactional correct projection, stable replication class, no transient unmanaged handoff | | Edge Reload/serve/RetireExcept and aliases/path routes | Shared manager predicate; one primary process; missing policy support fails closed | | Activator begin/wake/sweep/dead-child/adoption/due paths | Resident desired set, no idle stop, proactive bounded recovery, coalesced startup | | Scheduled token/control and sidecar paths | All local/durable fences apply; no artificial activity used for residency | | Placement/move/everywhere routes | New class cannot become multi-resident or enter old move worker accidentally | | Agent boot/fast/serial/root adoption, config sync/reap, restart generation/scopes, caps | Lifecycle-aware dispatch; no unit fallback or config changes while fenced | | Root litestream and hibernate restore-verification tools | Exactly one replication owner; resident exec included in correct lease-backed restore probe | | API/CLI response shapes, instance reports, status/schedule/metrics | Report non-hibernating but activator-owned residency correctly; no sleeping/stopped inference from a listening socket | | Maintenance preflight/guard/fence/resume and rollback-floor consumers | Updated explicit class; resident retry never supplies authority to clear a fence |
Acceptance requires these independent proofs:
- With zero traffic, timers and sidecars, a cold agent starts exactly one healthy process for each eligible resident primary. Advance through several idle windows: it remains awake. A legacy hibernating instance still sleeps and a legacy external route never gets spawned.
- Kill the resident app, adopt a new release, and issue an ordinary restart without traffic. Each recovers proactively with correct health/version. Repeated failures back off; requests and serial nudges cannot create a storm or bypass the ceiling. Failed cleanup blocks replacement.
- Concurrent request/due/reconcile/owner change/lease loss yields one lease and writer. Stale desired results cannot publish a process after retirement or changed owner. Incomplete registry/store discovery does not fabricate retirement or authorize fresh startup.
- The exact preflight-pinned active route account/app/host/creation identities and DB/WAL identity and semantic contents, Home principal IDs, directory generation, issuer keys, config/DEK/pepper sources and lease/replica prefix remain unchanged across activation, app restart and agent restart. Missing local DB restores only a validated existing replica; missing/corrupt replica never reaches Seed.
- Actual root replication configuration excludes both hibernating and resident exec DBs while preserving genuine unit coverage. Mutations using only
Hibernate, clearing replication on projection, or making all false-hibernate sockets managed must fail. - Policy schema/read/write tests cover every classified parser/writer family, including retry, delayed confirmation, raw list/purge, Upsert INSERT/CONFLICT rung preservation, replicationRail/carlos add, and former SetHibernate projection. Mutate each guard separately. A v1 row with stripped authoritative record must remain denied and root-excluded through repeated reconciliation and Edge.Reload, with checked cleanup and no external proxy handoff. Conflicting mode/owner/placement operations refuse before earlier side effects.
- Fence versus queued/in-flight resident startup, scheduled token preparation, stale Route, sidecar start and adoption leaves no unauthorized child or environment mutation. Actual retained handles are stopped; absence alone is not proof. Maintenance state cannot be retried as a crash.
- Restart the real canary agent supervisor with a descendant holding a pipe. Prove no old process tree survives to become a second writer. Hold a wake inside Start while Shutdown runs; return a live process afterward and prove joined checked cleanup before lease closure, with no awake publication. Check deadline expiry retains uncertainty. Existing unit/cgroup deployment capability and checked empty-tree evidence remain required.
- Exercise maintenance on a synthetic always-awake instance only after the full maintenance gate: checked stop, no resident reawakening, private operation, checkpoint/restore receipt, explicit resume with the selected-release non-serving checker, then sustained residency. Omitted, failed, stale, wrong-release or wrong-DB checker evidence refuses before serving even when policy receipt and HTTP health would pass. Failure, crash and ambiguous publication remain fenced.
- Aliases and path routing resolve the primary without duplicate startup. Credential and directory Home roles keep distinct state. Actual status reports process health/version and desired policy rather than treating socket presence as residency.
- Exact mixed-version/writer negative controls prove activation refuses when any required consumer or enforced floor is missing. An old-binary rollback cannot silently reenable hibernation, release ownership or introduce root replication.
- Crash at every fixed-action boundary: claim, guard, projection, stop, executing, policy CAS, receipt, registry transaction, succeeded, resume guard and resume projection. Prove the recovery table, no automatic policy replay/start, exact old/target comparison, no-field-loss mutations, distinct recovered receipt, fresh owner recovery/resume and refusal with uncertain tree/lease. Include the paused previous maintenance executor counterexample even when the Home process tree is empty and a new DB lease is obtainable; refusal must precede all repair/publish/start.
- A deployed synthetic canary, not localhost alone, proves cold startup, idle, crash, adoption and restored cold standby. Record observed recovery times and a written restore RTO. All native/race/cross/CI gates and independent Fable review pass before real Home policy changes.
Open decisions that block implementation approval
The choice of an explicit always-awake exec lifecycle and preservation requirements are this proposal. Review must settle the following before code:
- Accept or reject the first-slice outage consequence: an executor crash at any fixed-action boundary leaves Home fenced with no supported recovery until a separately reviewed exclusive-executor handoff exists. No fresh approval, elapsed time or new DB lease overrides this limitation.
- Approve or reject the fixed platform action, inert policy preparation, recovered-target receipt and pinned-owner fresh recovery lease rule above. They specify activation and retained-executor recovery without a generic claim enum, but are NEW authority proposals, distinct from app-exec. Fresh-lease takeover remains unavailable until the explicit exclusive-executor/death proof is implemented and reviewed; it is not implied by this design. Rejection leaves this branch not implementation-ready until revised.
- The selected Home release must carry at least one verified signed app runtime contract for ResumeRequested/Resumed and every later wake. Its reviewed signing/verification mechanism is a separate implementation gate from the authenticated platform binary floor.
- The authenticated platform capability/floor identifier and evidence needed for all old writers and the root replication generator. Its signed-floor foundation is still pending independent review.
- The human-owner policy endpoint and CLI spelling, including current-owner checks for a long-running convergence wait. This document proposes them; no existing API is claimed to authorize this field already.
- The four-start ceiling, retry timings and effective startup readiness deadline on the actual supported box. Preserve the stated bounds unless review changes them; measure an RTO rather than infer it from timers.
- A supported app-specific non-serving compatibility checker in each selected Home serving release. If none exists, implementing and reviewing it is a consumer dependency; current Home operator preparation, fixed-action success, recovered-target evidence and HTTP health are not substitutes.
- Exact Home primary/alias/role/config/DB ownership inventory from supported preflight. Supplied effective route receipts do not establish those facts.
Implementation must also test Daemon with Hibernate=false and ManagedByActivator=true: refuse before unit/file writes, including rollback/adoption paths. Repeat with Daemon=false but a granted port and assert no socket/service write or start; preflight must also refuse unknown grants.
No Home policy change, unit migration, maintenance operation, signing contract or runtime-floor declaration is approved by committing this design.