carlos / platform Public

Download this file

1	# Instance moves: two-phase, data-following (issue #331)
2
3	Status: DRAFT for review — 2026-08-28
4	Companion issues: #329 (region on create — shipped separately), #330
5	(replica_regions — shape only, untouched here), titogo residence design
6	§5/§8.3 (the consumer contract this serves).
7
8	## Decisions taken in conversation (Paul, 2026-08-28)
9
10	- Cross-bucket IAM is **push-only into a staging prefix** (`moves/`),
11	never into `instances/` — a box can stage state toward a sibling
12	region but can never touch another region's live replica lineage.
13	- The platform moves **the instance DB and record only**. The uploads
14	bulk-copy stays with the tenant (titogo's router), which owns the
15	bucket, the credentials, and the knowledge of which keys belong to
16	which instance. The move state machine exposes the phases so the
17	tenant can sequence its own copy inside the frozen window. If a
18	per-instance uploads prefix ever becomes a declared platform concept
19	(a record field), the move widens to carry it without unwinding this
20	design.
21
22	## The invariant
23
24	*Residence is a property of a route; the record's `region` field is the
25	residence carrier* (ingress spec §Anticipated #2). This spec never adds
26	a second arbiter: at every instant exactly one region's box is in scope
27	for the record, and the flip of `region` is the LAST act of the source.
28	Never two writers falls out of that plus the freeze the tenant holds
29	across the cut.
30
31	## Wire shape
32
33	The record gains one additive, omitempty object — present only while a
34	move is in flight:
35
36	```json
37	"move": {
38	"to": "uk",
39	"state": "preparing" | "ready" | "cutting" | "aborting",
40	"requested_at": "<RFC3339>",
41	"ready_at": "<RFC3339, when state >= ready>"
42	}
43	```
44
45	Records written before this field parse unchanged (wire-shape rule).
46	`move` is written by the console (the two verbs below) and by the
47	SOURCE box (the `preparing → ready` promotion and the terminal
48	clear-and-flip); each write is the narrow CAS `SetRecordRegion` already
49	uses. The record's `region` stays the SOURCE region until the cut
50	completes — every reader that knows nothing about moves keeps a
51	consistent answer throughout.
52
53	## The two verbs
54
55	Console API, on one instance, gated exactly like every other instance
56	door (app bearer; the tenant's router holds the token — never the
57	instance itself):
58
59	- `POST .../instances/{host}/move` body `{"region":"uk"}` — validated
60	against the armed-placement set (#329's gate, mandatory here: a move
61	to an unarmed region is always a 400, dormant never applies because a
62	move NEEDS a live target box). Refused 409 if a move is already in
63	flight, 400 if `to` equals the current region. Writes
64	`move:{to,state:"preparing"}`.
65	- `POST .../instances/{host}/move/cut` — refused 409 unless
66	`move.state == "ready"`. Writes `move.state:"cutting"`. The tenant
67	calls this only after freezing writes; the platform does not verify
68	the freeze (it cannot — the freeze is app-level), it documents it.
69	- `DELETE .../instances/{host}/move` — abort, legal in `preparing` or
70	`ready`. TWO-PHASE (review round 1): the door marks `aborting`; the
71	SOURCE box deletes the staged copy from the target bucket and then
72	clears `move` — a bare clear would leave staging behind to win a
73	later landing on the target (staging-wins below). Refused 409 in
74	`cutting` (completes forward) and in `aborting` (already under way).
75	- State is read, not pushed: the existing single-record GET already
76	returns the record verbatim, so `move.state` is pollable today with
77	zero new read surface.
78	- Every move write bumps `control/serial` (the config fast-convergence
79	nudge the env door already uses), so the source box notices in
80	seconds, not on the hourly floor. Between states the source's own
81	move worker self-paces its delta rounds.
82
83	## What the source box does
84
85	The reconcile/activator pass on the box whose `CARLOS_REGION` matches
86	`region` (the source — the record still points at it):
87
88	1. **preparing** (AMENDED during planning, 2026-08-28: the draft said
89	"a second litestream replica of the live DB" — two litestream
90	processes cannot tail one WAL, both want to own checkpointing. The
91	fallback IS the design, and it is better): the move copies the
92	REPLICA, never the database. The activator already keeps
93	`instances/<host>/` in the source's own bucket continuously synced
94	(running child) or final-checkpointed (parked child) — the move
95	worker mirrors that prefix into
96	`s3://<target-data-bucket>/moves/<host>/` with server-side
97	CopyObject (its IAM: read own bucket, push sibling staging), in
98	delta rounds every ~30s — List both sides, copy new/changed keys,
99	delete keys the source pruned (a true mirror, so litestream
100	retention holds in staging too). Uniform for running and parked
101	instances; no wake, no litestream internals touched.
102	Target bucket name and AWS region are derived, not configured
103	per-move: each box publishes its data bucket coordinates as two new
104	additive fields in its edges file (`data_bucket`, `data_bucket_region`
105	— same additive rule as `placement`), so the source reads the
106	target's coordinates from the same file set everything else reads.
107	2. **ready promotion**: once a delta round completes with zero
108	differences against a fresh listing, the source CASes
109	`move.state: "preparing" → "ready"`, stamping `ready_at`. Rounds
110	keep running — ready is a level, not an edge; writes after ready
111	keep flowing into staging on the next round.
112	3. **cutting** (order amended, review round 1): on seeing
113	`state:"cutting"`, the source (in order): set the route's Status to
114	"moving" (a non-ready status: the edge
115	stops serving it, the edges file drops it, and the activator's
116	RetireExcept sweep stops the child through the ordinary stop path —
117	which final-checkpoints to the source's OWN bucket — but SWALLOWS a
118	checkpoint failure, safe only while the warm cache stood behind it)
119	→ wait for the child to be down → **re-checkpoint the local DB with
120	the lease and wait** (proving the own replica carries the tail —
121	review round 1, the failed-final-checkpoint data-loss hole) → one
122	final delta round into staging (now byte-identical to the own replica) →
123	**delete the local DB, WAL/SHM, and litestream meta dirs** (the
124	warm-cache purge; without it a later move back would resurrect a
125	stale local file — the activator's warm-cache trust is exactly right
126	for crashes and exactly wrong here) → one CAS that sets
127	`region: <to>` and clears `move` → remove its own route row on its
128	next pass (the record has left its scope, the existing retirement
129	path does this unprompted).
130	If the box dies mid-cut, the record still says `cutting` with the old
131	region: the source resumes the sequence on restart (every step is
132	idempotent — sync again, stop a stopped child, delete absent files,
133	CAS already-flipped is a no-op conflict it re-reads).
134
135	## What the target box does
136
137	Nothing move-specific until the flip — then its ordinary pass sees a
138	record newly in scope and mints the route. One addition in the
139	activator's `Restore`: when the local DB is absent AND
140	`instances/<host>/` in its own bucket has no replica, it checks
141	`moves/<host>/` (own bucket, own IAM); if present, server-side copy
142	`moves/<host>/ → instances/<host>/`, restore from there, `CheckDB` as
143	always, and delete `moves/<host>/` on success. A corrupt staged replica
144	quarantines exactly like a corrupt own one — and the abort story still
145	stands, because the source's last own-bucket checkpoint is intact and
146	`region` can be pointed back by hand (`SetRecordRegion` — the operator
147	recovery door that already exists).
148
149	## IAM (the infra half — platform-infrastructure + tito repo applies)
150
151	`region-host` (and `region-remote-host`) gain an optional
152	`sibling_staging_arns` list; when set, the role gains `s3:PutObject` on
153	`<each sibling data bucket>/moves/*` — and nothing else: no Get, no
154	List, no Delete, never `instances/*`. Residency-at-rest holds
155	(cross-region reads remain impossible at IAM); the write blast radius
156	is confined to staging keys no box ever trusts as live state without
157	its own restore-time `CheckDB`. The mirror needs three actions on the
158	sibling, all `moves/*`-scoped: `s3:PutObject` and `s3:DeleteObject` on
159	the object ARNs, and `s3:ListBucket` on the bucket ARN with an
160	`s3:prefix` condition of `moves/*` — still no reads of sibling data,
161	still nothing outside staging.
162
163	## Instance env
164
165	Already shipped with #329: every exec child gets `CARLOS_REGION` (the
166	box's placement code), agent-owned in both directions. An app that
167	lands on the target box after a move reads the new value at boot —
168	titogo's boot-clears-freeze rule needs exactly this.
169
170	## Out of scope
171
172	- Uploads (tenant-side, by decision above).
173	- `replica_regions` behavior (#330 stays shape-only).
174	- Moving unit-backed, addr-backed, static, or everywhere-class routes:
175	the verbs refuse anything but an exec-backed `kind:"instance"` route
176	(400 naming the kind) — the only class whose state the platform owns.
177	- Fleet placements (`fleet/<name>`): refused; fleets keep D6
178	state-does-not-follow semantics on purpose.
179	- Pooled records (`pool != ""`): refused in v1 — their data lives on
180	the pool's box, not the platform region box that would drive the
181	move. Non-hibernating instances (`hibernate:false`): refused — they
182	run outside the activator, so nothing manages (or could quiesce)
183	their data.
184	- The worker acts only on routes its OWN registry serves: region
185	equality alone would let a standby or second same-region box drive a
186	cut whose child runs on its sibling (per-box registries make its
187	Quiesced trivially true).
188	- Automatic freeze verification; region-aware unit economics; the
189	member-facing residence door (titogo builds on these verbs).
190
191	## Testing
192
193	- Record CAS: move-field transitions (prepare/ready/cut/abort/conflict)
194	against the fake store, including the 409 matrix.
195	- Source pass: fake-store + fake-litestream harness proving the cut
196	ordering (final sync BEFORE stop BEFORE local purge BEFORE flip), the
197	parked-instance prepare (no wake), resume-from-`cutting` idempotence,
198	and that abort leaves the source serving with its own replication
199	intact.
200	- Target restore: staging fallback (copy, restore, CheckDB, cleanup),
201	corrupt-staging quarantine, and precedence (own `instances/` replica
202	beats staging when both exist — a re-landed instance must not regress
203	to an older staged copy).
204	- Console: verb gates (bearer, kind, armed region, state machine),
205	poll-through-GET.
206	- The never-two-writers argument pinned by an ordering test: a request
207	hitting the source during `cutting` after stop is a 503 from the edge
208	(route gone on next pass) while the target has not yet minted. The
209	length of that window is the target's reconcile latency, NOT assumed:
210	measure it during build (the adopt path/timer cadence is the floor)
211	and record it in the runbook — the tenant's freeze must be budgeted
212	to cover it, and if it measures in minutes rather than seconds, a
213	serial-bump nudge on the record write is the follow-up, not a redesign.
214