amadan
spec: region HA spec 1, placing an instance on a box (rev 1)
Rulings H1-H6 (2026-10-04), the claim naming any eligible box, relocation and the termination fence, and the changes requested of the claim and moves specs (K1-K9).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+439
−0
docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md+439 −0
CI failed — run details
diff --git a/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md b/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md| new file mode 100644 |
| index 000000000..6feb48a85 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md |
| @@ -0,0 +1,439 @@ |
| +# Region HA, spec 1: placing an instance on a box |
| + |
| +Design, 2026-10-04, **rev 1**. DRAFT. Nothing here is built. |
| + |
| +This is the first of four specs that give each region high availability |
| +and autoscaling: |
| + |
| +1. **Box placement** (this spec). With several boxes in a region, the |
| + host lifecycle claim names the one box that runs each instance, and |
| + every box, edge and activator follows the claim. |
| +2. **Placement policy.** Which eligible box a new or displaced instance |
| + goes to. |
| +3. **Failover.** A dead box's instances are re-placed onto the region's |
| + survivors. |
| +4. **Autoscaling.** A region's boxes are an Auto Scaling group that grows |
| + on demand and shrinks by draining. |
| + |
| +It builds on two specs still in review, and asks them for changes (§9): |
| + |
| +- the host lifecycle claim, rev 6.3 (`design-replica-ownership`, |
| + 8faa89a4), cited as **claim §n**; |
| +- move-any-app, rev 7.5 (`design-moves-any-region`, cdc98845), cited as |
| + **moves §n**. |
| + |
| +It also builds on nearest-region (on main since 3f4a9bfd), whose box |
| +evidence in the edges file is how a console knows which boxes a region |
| +has. |
| + |
| +## 0. Rulings (Paul, 2026-10-04) |
| + |
| +| # | Ruling | |
| +|---|---| |
| +| H1 | **The proof that a box is dead is that AWS terminated it.** Each region's boxes run in an Auto Scaling group with health checks. When the group terminates an unhealthy box, a region placer that has only read-only EC2 and Auto Scaling rights sees `terminated` and re-places the box's instances. The console still holds no EC2 rights. This replaces M16 / claim §7.2's operator-only declare-lost **for terminated boxes only**; a `stopped` box stays manual. It also replaces fleets D7's "no automatic reaper" for platform boxes. Lease expiry is still never proof (CC5). | |
| +| H2 | **Shape the claim now.** The HA specs cite the claim rather than invent a second primitive, and send the claim and moves sessions the exact changes they need (§9) before those specs are finalised on a one-box-per-region assumption. | |
| +| H3 | **About one second of data loss, written down.** Litestream keeps shipping asynchronously. Writes that had not reached S3 when a box died are lost, and the docs and console say so. | |
| +| H4 | **Placement policy: capacity plus soft spread.** Among eligible boxes, choose the one with the most headroom. Prefer a box that runs no other instance of the same app, but never refuse placement for that reason. If no box is eligible, the instance waits as `unplaced` and an operator is alerted. (Spec 2.) | |
| +| H5 | **Each region's minimum is configurable: 1 by default, 2 where HA is sold.** Scale-out is automatic on low headroom. Scale-in is automatic too, but only after the box has been drained into the region's other boxes. (Spec 4.) | |
| +| H6 | **File durability follows the paid plan.** Continuous sync on the top plan, periodic snapshots on the bottom one. The platform exposes this as a per-app setting (§5.6) and Carloku maps its plans to it. Platform-managed files (`instance.key`, `report.token`, `<host>.env`) sync continuously on every plan. | |
| + |
| +Standing rulings this spec keeps: |
| + |
| +- One writer: a home-run instance runs on exactly one box (claim C4, |
| + multi-region ingress, sidecar members). |
| +- Placement targets a region and pool, never a named box (fleets D5). |
| + An operator or a customer still cannot pin an instance to a box; only |
| + the placer chooses. |
| +- The console is the single control-plane writer and holds no EC2 rights |
| + (claim §7.2). |
| +- A box must hold the deployment's secrets key to receive instances (R9), |
| + and every box reconcile would admit in a region must qualify |
| + (nearest-region F2). |
| +- Unreplicated state on a dead box is lost, and we say so. |
| + |
| +## 1. Why |
| + |
| +Today a region is one box. Placement is per region: a bucket record has |
| +`Region` and `Pool` and no box (`internal/provision/provision.go:39`). |
| +Every box whose `CARLOS_REGION` matches mints the route |
| +(`internal/provision/reconcile.go:335`). Nothing enforces "exactly one |
| +box" (`reconcile.go:179`). |
| + |
| +A second box in a region would break three things: |
| + |
| +1. **Both boxes run the instance's route.** The S3 lease (`lock.json`, |
| + 15-minute TTL, `internal/lease/lease.go:38`) stops them running at |
| + once, but whichever box a request reaches wins. |
| +2. **Both boxes claim the host to the edges.** The home map breaks the |
| + tie by sorted box name, with no liveness (`internal/edge/homemap.go:585`), |
| + so a dead box that sorts first keeps winning. |
| +3. **A box can serve a stale local copy.** The activator trusts a local |
| + database as the warm cache because "on one box … the local copy can |
| + only be what we ourselves checkpointed" (`internal/activator/activator.go:1127`). |
| + With two boxes taking turns, that is false. |
| + |
| +And when a box dies, its instances stay down. EC2 auto-recover brings |
| +back the same machine, if it can (`infra/modules/region-host/main.tf:497`). |
| + |
| +## 2. Words |
| + |
| +- **Box**: one platform host machine. Not a customer fleet box; fleets |
| + keep their own mechanism and are out of scope. |
| +- **Box id**: the box's EC2 instance id (`status.InstanceID()`, |
| + `internal/status/instanceid.go:37`). Instance ids are never reused, so |
| + a box id names one machine for ever. Boxes that predate this spec keep |
| + their label (`status.Label(machine-id)`) and publish their instance id |
| + beside it (§5.1). |
| +- **Executor**: the one box the claim's `op.exec` names for a host. |
| +- **Eligible box**: a box that is fresh (nearest-region F1), reachable, |
| + hosting, serves the instance's pool, accepts inbound, holds the secrets |
| + key (R9), is not draining, and is not on the claim's `lost` list. |
| +- **Region placer**: one box per region, chosen by the existing lease |
| + election, that runs re-placement (spec 3) and drain (spec 4). Its |
| + election lease only decides who acts, never what is true. |
| +- **Relocation**: changing a host's executor to another box **in the |
| + same region**. A move changes region; relocation does not. |
| + |
| +## 3. Invariants |
| + |
| +**HA1. The claim names the executor.** A home-run host (exec, unit, |
| +socket, sidecar) runs only on the box its claim's `op.exec` names. This is |
| +claim C4 unchanged. What changes is that `op.exec` may now name any |
| +eligible box in the record's region, not "the region's box". |
| + |
| +**HA2. Region equality never admits.** Reconcile, the activator, config |
| +sync, release adoption and timers act for a host only on its executor. |
| +Being in the right region is necessary, never sufficient (claim C4 says |
| +this already; today's code does not do it). |
| + |
| +**HA3. A local copy is trusted only under the current installation.** A |
| +box may skip restore and serve its local database and files only when its |
| +local stamp record has `active_stamp == holder.stamp` from a fresh claim |
| +read (claim §6.4). Otherwise it restores from S3. This replaces the |
| +one-box warm-cache argument. |
| + |
| +**HA4. Edges follow the newest installation.** When two edges files claim |
| +one host, the edge believes the one carrying the higher `holder.stamp`. |
| +Stamps only increase (claim `gen`), so a dead box's stale file always |
| +loses to its successor's, with no liveness rule. |
| + |
| +**HA5. Everything an instance needs to start is in S3.** The database, |
| +its files directory and its env file are under the incarnation's |
| +`holder.location`, or are rendered from the record (config, release). |
| +Nothing needed to start an instance lives only on the box running it. |
| + |
| +**HA6. The executor changes only by a claim operation.** Create, a move, |
| +a relocation (§5.8). Each changes `op.exec` in a CAS that cites stop |
| +evidence (claim §6.1) or a termination fence (§5.9), and installs a new |
| +generation with a new stamp. |
| + |
| +## 4. What stays the same |
| + |
| +- A region with one box behaves exactly as today, and every new rule |
| + holds trivially there. |
| +- Static sites and public addr routes still run on every box |
| + (`everywhereClass`, `reconcile.go:195`). Only home-run hosts gain an |
| + executor. |
| +- Customer fleets keep their binding stamps and per-request fallback |
| + (`internal/edge/fleetbind.go`). |
| +- The ingress model: any edge serves any app and backhauls home-run |
| + hosts to their box (multi-region ingress). This spec changes which box |
| + is "home", not how backhaul works. |
| +- Latency steering and ingress for boxes that come and go are spec 4. |
| + |
| +## 5. Design |
| + |
| +### 5.1 Box identity and evidence |
| + |
| +Each box's edges file (`internal/host/edgesfile.go:157`) gains: |
| + |
| +```json |
| +{ |
| + "box_id": "i-0abc…", |
| + "draining": false, |
| + "headroom": { "mem_free_mib": 2210, "disk_free_mib": 31200, "awake": 14 } |
| +} |
| +``` |
| + |
| +- `box_id` is the EC2 instance id from IMDS, or the label on a machine |
| + with no IMDS (dev, Pi). A box launched by an Auto Scaling group uses its |
| + instance id as its label too, so the two are the same. |
| +- `draining` is set by the placer through the box (spec 4), and an |
| + eligible-box check refuses a draining box. |
| +- `headroom` is the input to spec 2's choice. Spec 1 publishes it only. |
| + |
| +The console already reads these files with freshness |
| +(`internal/console/instancesplacement.go:93`). Nearest-region's |
| +`regioncandidates.go` keeps deciding which **regions** are eligible. |
| +This spec adds `eligibleBoxes(snapshot, region, pool)`, which returns |
| +the boxes, applying the same predicate one box at a time (nearest-region |
| +`TestEligibilityNeverCombinesBoxes`). |
| + |
| +### 5.2 Create names a box |
| + |
| +Claim §5.1 step 4 becomes: CAS `op` to `serving` and set `exec` to |
| +`choose(eligibleBoxes(region, pool), app)`. |
| + |
| +- Until spec 2 lands, `choose` is: the eligible box running the fewest |
| + claimed hosts, preferring one that runs no other instance of the app, |
| + ties broken by box id. It is a pure function over the snapshot and the |
| + claims the console already holds, and it is unit-tested alone. |
| +- No eligible box: the create fails with today's "region has no box |
| + that can take this" refusal (nearest-region). Spec 2 adds the |
| + `unplaced` state for re-placement, where refusing is not possible. |
| +- The choice is recorded only in the claim. The record keeps `Region` |
| + and gains no box field, so a record still describes residency, not |
| + placement. |
| + |
| +### 5.3 Reconcile admits only on the executor |
| + |
| +`inScope` (`reconcile.go:335`) keeps its pool and region test, and for |
| +home-run classes adds a fresh claim read: mint or keep the row only if |
| +`op.exec` names this box (claim C4, "reconcile's decision to create or |
| +keep a row"). |
| + |
| +- The safety gate is the activator's fresh claim read at every spawn |
| + (claim C4), not reconcile. So reconcile reads a host's claim only when |
| + it would **create** a row, and re-checks the rows it keeps on a slow |
| + sweep (every 10 minutes). Reading every claim on every pass would cost |
| + real money: 2,000 hosts every 30 seconds is about 5.8 million GETs, or |
| + about $2.30, a day per box. |
| +- A relocation tells both boxes directly: the placer bumps the region's |
| + control serial after each executor CAS, so reconcile runs at once and |
| + re-reads the claims of hosts it holds or would create. |
| +- A claim read error keeps an existing row and creates none (claim C6). |
| + A box that cannot read claims keeps serving what it already serves. |
| +- A row whose claim names another box is released through the claim's |
| + stale-hold rule (claim §8, CC10), never by deleting local state: |
| + retirement is the operation's job (§5.8). |
| + |
| +### 5.4 Edges and the home map |
| + |
| +- The edges-file writer (`edgesfile.go:185`) publishes a host as `home` |
| + only when this box is its executor, and adds the installation's stamp: |
| + `"hosts": {"shop.bab.oncarlos.com": {"role": "home", "stamp": 41}}`. |
| +- The home map (`homemap.go:529`) resolves a host claimed by two files to |
| + the higher stamp (HA4). Equal stamps can only come from two legacy files |
| + with no stamp; those keep today's sorted rule, and an alert fires |
| + because it means two boxes are serving one host. |
| +- The edge never reads claims on the request path. |
| + |
| +### 5.5 The activator's local copy |
| + |
| +The warm-cache path (`activator.go:1127`, `backend_exec.go:265`) checks |
| +HA3 before trusting a local database: under the host guard, it reads |
| +the claim fresh, compares `holder.stamp` with the box's `active_stamp`, |
| +and restores from S3 on any mismatch, including a missing local record. |
| +The comment's one-box argument is replaced with HA3. |
| + |
| +### 5.6 Files reach S3 continuously or by snapshot |
| + |
| +Moves carry the files directory (F4) and `<host>.env` (F6) as a sealed |
| +archive taken from the live source (moves §6.1, §8). A dead source can't |
| +produce one, so HA5 needs them in S3 already. |
| + |
| +**The setting.** An app's manifest gains `files_durability`: |
| + |
| +- `continuous`: each changed file is uploaded within about one second of |
| + its last write (debounced on close and rename, via inotify on the |
| + host's directory, with a full listing every 10 minutes to catch |
| + anything a watch missed); |
| +- `snapshot`: a sealed archive of the whole directory every |
| + `files_snapshot_interval`, default 15 minutes. |
| + |
| +Carloku maps its plans to this (H6). The default for an app with no |
| +setting is `snapshot`. |
| + |
| +**Always continuous, whatever the setting:** `instance.key`, |
| +`report.token` and `<host>.env`. They are small and rarely change, and |
| +losing them is what kept kpmg down for 1h54m (moves §1.2). |
| + |
| +**Where.** Under the incarnation's own location, inside the current |
| +generation (claim C3a, moves §5.1): |
| + |
| + <holder.location>/files/<generation>/<relative path> (continuous) |
| + <holder.location>/files/<generation>/snapshots/<seq>.tar (snapshot) |
| + |
| +Both buckets are versioned, so a restore names exact versions (claim C3). |
| +A delete in the files directory is uploaded as a tombstone object, not |
| +an S3 delete. |
| + |
| +**Restore.** A box that becomes executor restores the database, then the |
| +files: the latest snapshot, then any continuous objects newer than it. |
| +The files and the database can disagree by up to the snapshot interval |
| +on a `snapshot` app, and the console says so on the app's Overview. |
| + |
| +**Moves.** A move keeps its sealed archive at the cut (moves §8). The |
| +continuous objects are an addition, not a replacement, so moves is |
| +unchanged. |
| + |
| +**Sidecars.** `<host>-sidecar.env` (F8) is treated like `<host>.env`. |
| +Moves refuses sidecar apps (M10); relocation (§5.8) does not need to, |
| +because nothing crosses a region. |
| + |
| +### 5.7 What a new executor rebuilds |
| + |
| +| State | How the new box gets it | |
| +|---|---| |
| +| Database | restore from `holder.location`, current generation | |
| +| Files directory, env files | §5.6 | |
| +| `<host>.config.env` | rendered for the held row (moves §9.1), needs the secrets key (R9) | |
| +| Release | materialized for the held row (moves §9.2) | |
| +| Timers (`next_due`) | recomputed from the schedule at the first reconcile. A tick due during the gap runs once on the new box; a tick that ran on the dead box but whose result was not replicated may run again. Ticks are documented as at-least-once. | |
| +| Certificates | the cert mirror's boot refresh (on main) | |
| +| Registry row | reconcile, once the claim names this box | |
| +| Inventory | written by the new box before it first materializes state (claim §6.3) | |
| + |
| +### 5.8 Relocation: the one operation that changes the box |
| + |
| +A new claim op kind, `relocating`, from `serving` only, same region only. |
| +Its intent names `{from, to, reason}`, where reason is `drain`, |
| +`rebalance` or `failover`. |
| + |
| +1. **Start CAS:** `op = {relocating, id, intent, exec: from, participants: |
| + [from, to]}` (or `participants: [to]` for failover). Both write |
| + execution leases (claim §7.1) before leaving the first phase. |
| +2. **Release the source**, by one of: |
| + - **stop evidence** from `from` (claim §6.1), after a final |
| + replication flush, for drain and rebalance; or |
| + - a **termination fence** (§5.9), for failover. |
| +3. **Install on the target.** CAS `exec: to` with a new |
| + `holder.generation` and `holder.stamp` (claim §6.4). Late writes from |
| + the old installation land in the old generation, which nothing |
| + admitted reads (claim C3a, within one incarnation per moves §5.1). |
| +4. **Target restores** (§5.7), takes the replica lease, and the host goes |
| + back to `serving` once the target's probe passes (moves §7.7's probe, |
| + same region). |
| +5. **Source parks** its local state with `retire-installation` (claim |
| + §6.4), as a `park-local` obligation on `from`. A terminated box has no |
| + disk; the fence removes that obligation (claim §7.2's second case). |
| + |
| +**Recovery owner:** the region placer, from `intent`. It is a box, not |
| +the console, because failover must work when the console's box is the |
| +one that died. |
| + |
| +**Cost of a drain:** one cut of a few seconds, like a move's (M2), with |
| +no cross-region copy. Failover's gap is detection plus restore (spec 3). |
| + |
| +### 5.9 The termination fence |
| + |
| +A termination fence is a JSON object written into the operation's |
| +journal: |
| + |
| +```json |
| +{"box": "i-0abc…", "state": "terminated", |
| + "observed_at": "<RFC3339>", "via": "ec2:DescribeInstances", "region": "eu-west-1"} |
| +``` |
| + |
| +- Only `terminated` counts. `stopped` and `stopping` do not, because a |
| + stopped box can start again with its disk. A stopped box still needs |
| + the operator's `carlos ops host resolve --declare-lost` (claim §7.2). |
| +- The placer reads the state itself with its own read-only credentials |
| + (§6). It never trusts a box's own report, an edges file, or the Auto |
| + Scaling group's lifecycle event alone; the event only wakes it. |
| +- A terminated instance cannot run again, so a terminated box needs no |
| + quarantine or `--readmit`. Its id goes on `lost` anyway, so a forged or |
| + confused file naming it is refused. |
| +- The root volume must have `DeleteOnTermination`, so no disk outlives the |
| + box (§6). |
| +- Late requests: once EC2 reports `terminated`, the machine sends |
| + nothing more. A request S3 had already accepted may still complete for |
| + a few seconds. It lands in the old generation (§5.8 step 3) and counts |
| + toward the H3 loss window. |
| + |
| +## 6. Infrastructure and IAM |
| + |
| +- Each box's role gains `ec2:DescribeInstances` and |
| + `autoscaling:DescribeAutoScalingGroups`, conditioned on its own region |
| + and the deployment's tag. No `ec2:Stop*`, `Terminate*` or `Modify*`. |
| + The console's role is unchanged (no EC2 rights). |
| +- Boxes need `s3:PutObject` and `s3:GetObjectVersion` on |
| + `instances/<host>/*/files/` in their region's data bucket (§5.6), and |
| + the claim CAS rights the claim spec already lists. |
| +- The `region-host` module gains `delete_on_termination = true` on the |
| + root volume. Moving to an Auto Scaling group is spec 4; spec 1 runs on |
| + today's fixed boxes and on a second fixed box added by hand. |
| + |
| +## 7. Failure modes |
| + |
| +| What happens | Result | |
| +|---|---| |
| +| Executor box dies, region has another box | spec 3 relocates its hosts after the termination fence. Until then: requests 503 with Retry-After. | |
| +| Executor dies, region had one box | the Auto Scaling group launches a replacement; it wins the placer election, sees the old box terminated, and relocates every host to itself. | |
| +| Box partitioned from the control bucket | it keeps serving what it serves (no execution lease while `serving`, claim §7.1). It can't take new work. The group's health check decides whether it is terminated. | |
| +| Box alive but failing its health check | the group terminates it, then as above. Instances that were healthy lose up to H3's window. | |
| +| Two files claim a host | the higher stamp wins (HA4). Equal stamps alert. | |
| +| Claim read fails on reconcile | keep existing rows, create none (C6). | |
| +| `snapshot` app fails over | files lose up to the interval; the database loses about a second. Stated on Overview. | |
| +| Placer box dies mid-relocation | the next elected placer replays from `intent` (§5.8). | |
| + |
| +## 8. Rollout |
| + |
| +1. **Prerequisite:** the claim and its floor are live (claim §10), and |
| + every host in the region has a claim (adopted under claim §9). |
| +2. Ship §5.1 to §5.7 dormant behind `CARLOS_REGION_HA`. With one box per |
| + region, every rule is a no-op; ship it to all four regions and watch. |
| +3. **A second box may join a region only when every home-run host there |
| + has a claim** (§9, K5). The join check is in `carlos system` and in the |
| + console's box list. |
| +4. Rehearse on dev with two fixed boxes: create spreads across both; |
| + drain one (§5.8) with `moveprobe` shaped like ips (moves §14.4); |
| + terminate one by hand and run the spec 3 placer manually. |
| +5. Then specs 2, 3, 4. |
| + |
| +## 9. Changes requested of the claim and moves specs |
| + |
| +For platform-1e (claim) and platform-d8 (moves). Each is needed before |
| +those specs finalise a one-box-per-region assumption. |
| + |
| +| # | Spec | Change | |
| +|---|---|---| |
| +| K1 | claim §5.1 step 4 | `exec` is the box `choose()` returns among the region's eligible boxes, not "the record region's box". | |
| +| K2 | claim §4, §8 | New op kind `relocating` (§5.8): same region, from `serving`, recovery owner the region placer. Excludes moves and placements, as `placing` does. | |
| +| K3 | claim §7.2, CC5 | A **termination fence** (§5.9) is a lost-box fence, written automatically by the region placer. `terminated` only; `stopped` stays operator-only. No `--readmit` needed for a terminated box; it still goes on `lost`. | |
| +| K4 | claim §8 | Region placers (boxes) need claim CAS rights for `relocating` and for removing a terminated box's obligations. | |
| +| K5 | claim §9 | Adoption names "the record region's serving box". With several boxes that is ambiguous, so adoption must finish while the region has one box; a second box joins only after (§8 step 3). | |
| +| K6 | claim §2 | Box names in `op.exec`, `participants`, `touched.boxes`, `inventory` and `lost` are box ids (§2); legacy labels stay valid. | |
| +| K7 | moves §7.10 | A move whose participant is terminated mid-move takes the termination fence as M16's evidence automatically, under the same before-commit rules. Moves decides what the placer may do there; the placer only supplies the fence. | |
| +| K8 | moves §5.1, §6.1 | The restore contract includes `files/<generation>/` (§5.6). F4 and F6 are also carried continuously; the archive at the cut stays. | |
| +| K9 | joint model | `internal/sim/hostlife` extends to two boxes per region: create spread, relocation by drain, termination mid-serve and mid-relocation, a stale edges file, a stale warm copy, late PUTs after termination. | |
| + |
| +## 10. Testing |
| + |
| +- **Unit:** `eligibleBoxes` (one box at a time, staleness, draining, |
| + `lost`); `choose` (spread, ties); home-map stamp resolution; HA3 |
| + restore decision; files tombstones and restore order. |
| +- **Integration:** two agents against one fake region, one control |
| + bucket: create lands on the chosen box only; the other box mints no |
| + row; the edge backhauls to the executor; relocate by drain; a stale |
| + local copy on the old box is not served after it becomes executor |
| + again. |
| +- **Joint model (K9)** decides readiness, as it does for the claim and |
| + moves. Prose rounds alone have stopped converging on this kind of |
| + protocol. |
| +- **Rehearsal** (§8 step 4). |
| + |
| +## 11. Open questions |
| + |
| +1. **Timers at-least-once (§5.7).** Is documenting ticks as |
| + at-least-once acceptable, or does a tick need an idempotency key |
| + carried in the claim? |
| +2. **`files_durability` default.** `snapshot` for apps with no setting, |
| + so existing apps pay nothing new until Carloku maps plans. Agreed? |
| +3. **The reconcile sweep (§5.3)** re-reads every kept host's claim every |
| + 10 minutes. If that's too slow to notice a stray row, spec 2 adds a |
| + per-region claim index; spec 1 doesn't. |
| + |
| +## 12. Out of scope, for the later specs |
| + |
| +- **Spec 2:** `headroom` thresholds, soft spread across an app's |
| + instances, the `unplaced` state and its alert, rebalancing. |
| +- **Spec 3:** the placer loop: election, Auto Scaling event wake-up, |
| + the termination fence, relocating every host off a terminated box in |
| + bounded parallel, waking always-on and scheduled instances, the |
| + failover gap target. |
| +- **Spec 4:** the Auto Scaling group per region, launch template and |
| + join without SSM hand steps, ingress for boxes that come and go (a |
| + per-region NLB with fixed EIPs, or an EIP pool claimed at boot), |
| + HTTP-01 certificates, steering, scale-out on headroom, scale-in by |
| + drain (§5.8), per-region minimum (H5). |