carlos / platform Public

branch:

Clone

git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1	# moveprobe
2
3	moveprobe is a small app for rehearsing instance moves. It keeps state in
4	every place a real app can, so that after a move you can check that each kind
5	of state arrived. It is not part of the platform release. You build it and
6	ship it like any other app.
7
8	## What it keeps
9
10	- Database rows. A background loop writes a row every half second. A
11	separate worker process writes one every second. Each timer fire writes one.
12	Every row records the region it was written in.
13	- A key file. On first boot it writes `instance.key` into the instance's
14	files directory and records the same key in the database. Every boot checks
15	that the two still match.
16	- The platform's instance key. On first boot it records a fingerprint of
17	`CARLOS_INSTANCE_KEY`. Every boot checks that the key is still set and
18	unchanged.
19	- An object in its store. On every boot it writes `probe/<boot id>.txt`
20	to the store bound with the prefix `MOVEPROBE`, and reads it back on every
21	readiness check.
22	- A sealed secret. It needs `MOVEPROBE_SECRET` to be set.
23	- Timers. A once-timer you arm through the app, and a schedule you set
24	with `carlos schedule set`.
25
26	## Build
27
28	The rehearsal boxes are arm64:
29
30	```sh
31	CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o moveprobe ./cmd/moveprobe
32	```
33
34	## Deploy
35
36	Run these signed in to the ops account, unless the rehearsal names another.
37	Use the deployment's instance domain where these say `example.com`, and pick
38	a host name no earlier rehearsal has used.
39
40	```sh
41	carlos ship --app moveprobe --label "moveprobe" ./moveprobe
42	carlos instances enable --app moveprobe
43	carlos store create --app moveprobe --env-prefix MOVEPROBE
44	carlos secrets set --app moveprobe MOVEPROBE_SECRET=<a long random value>
45	carlos schedule set --app moveprobe --name every-minute --every 1m --path /tick
46	carlos instances create --app moveprobe --host mp1.example.com --region ie
47	```
48
49	Leave private storage off when you enable instances, and create the store
50	without `--environment`. A move only carries the app's own store. It refuses
51	an instance whose store belongs to an environment or to the instance itself.
52	The account must be approved for object storage before `store create`
53	provisions anything.
54
55	Keep the value of `MOVEPROBE_SECRET` in your shell. It is also the password
56	for `/arm-once` and `/report`.
57
58	## Use it
59
60	```sh
61	curl https://mp1.example.com/ready
62	curl -X POST -H "Authorization: Bearer $MOVEPROBE_SECRET" "https://mp1.example.com/arm-once?in=600"
63	curl -H "Authorization: Bearer $MOVEPROBE_SECRET" https://mp1.example.com/report
64	```
65
66	\| path \| who can call it \| what it does \|
67	\|---\|---\|---\|
68	\| `/` and `/ready` \| anyone \| 200 when the key file, the platform key, the stored object and the secret all check out, and the worker's writes are landing. Otherwise 500 with every failed check. The store check runs at most once every 5 seconds, and requests in between share its result. A move probes `/`. \|
69	\| `GET /report` \| `MOVEPROBE_SECRET` \| JSON: row counts by kind and region with their lowest and highest sequence numbers, the timer fires by name and due time, both key fingerprints, the stored object, whether the secret is set, the worker's pid and failed writes, and the region the app saw. Any failed database read makes it a 500, never a partial report. \|
70	\| `POST /arm-once?in=<seconds>` \| `MOVEPROBE_SECRET` \| Arms a once-timer named `once-<n>` that calls `/once` after that many seconds. Answers 503 when the instance has no control socket. \|
71	\| `POST /tick`, `POST /once` \| the platform \| Timer fires. Each one writes a row with the timer's name and due time. \|
72
73	The background rows are numbered in one sequence that carries on across
74	boots and regions. If no row is missing, the count of `bg` rows equals the
75	highest `bg` sequence number.
76
77	The region comes from `CARLOS_REGION`. On a box that has no region, the app
78	uses `home`, and `/report` says so.
79
80	## Make readiness fail on purpose
81
82	`MOVEPROBE_FAIL_READY_IN=<region>` makes the app fail its readiness check in
83	that region. The app reads it when it starts, so set it before you request
84	the move, and remove it once the run's checks are recorded:
85
86	```sh
87	carlos env set --app moveprobe MOVEPROBE_FAIL_READY_IN=ie
88	carlos env unset --app moveprobe MOVEPROBE_FAIL_READY_IN
89	```
90
91	There are two modes:
92
93	- `MOVEPROBE_FAIL_READY_IN=ie` writes nothing at all in `ie`. It does not
94	open the database, touch the files directory, start the worker or call the
95	control socket. Every path answers 503. A rollback from here should be
96	clean.
97	- `MOVEPROBE_FAIL_READY_IN=ie,after-write` writes one background row in
98	`ie`, starts the worker, and then answers 503 on `/` and `/ready`. A
99	rollback from here is dirty, and the source should end up holding that
100	row.
101
102	In any other region the variable has no effect.
103
104	## The worker
105
106	The app starts a worker process in its own session, with no connection to
107	the app's output. Stopping the app does not stop the worker. That is the
108	point: it shows whether a move's stop really stops every process the
109	instance started. Once the app has gone, the worker keeps writing for 60
110	seconds and then exits by itself.
111
112	The worker tells the app whether each write landed. `/report` shows how many
113	committed and how many failed since the app started, how many of the latest
114	failed in a row, the last error, when it last heard from the worker, and
115	why the worker is gone if it is. `/ready` fails until a worker row has
116	committed since the app started, whenever the worker's latest write failed,
117	if the worker has sent nothing for 3 seconds, and once the worker has
118	exited.
119
120	## Strings
121
122	Every body the app can send, and when. Text in angle brackets is filled in
123	when the app answers. `/ready` joins its failed checks with `; `.
124
125	On `/` and `/ready`:
126
127	- `ready`: every check passed (200).
128	- `before-write mode: this box writes nothing`: the failure switch names
129	this region with no suffix. Every path gives this answer (503).
130	- `after-write mode: not ready here`: the failure switch names this region
131	with `,after-write` (503).
132	- `key mismatch: instance.key exists but the database has no key`: the
133	files directory has a key file, but the database never recorded one
134	(500).
135	- `key mismatch: instance.key is missing`: the database has a key, but the
136	file is gone (500).
137	- `key mismatch: instance.key differs from the database`: the file and the
138	database hold different keys (500).
139	- `key mismatch: instance.key: <error>`: the file is there but cannot be
140	read (500).
141	- `CARLOS_INSTANCE_KEY is not set`: the platform key is missing (500).
142	- `CARLOS_INSTANCE_KEY differs from the one seen at first boot`: the
143	platform key changed (500).
144	- `MOVEPROBE_SECRET is not set`: the sealed secret is missing (500).
145	- `store: <names> not set`: one or more of the store variables is missing
146	(500).
147	- `store: put <key>: <error>`: the boot's object could not be written
148	(500).
149	- `store: get <key>: <error>`: the boot's object could not be read back
150	(500).
151	- `store: <key> reads back different bytes`: the object came back changed
152	(500).
153	- `store: <error>`: the request gave up while waiting for a store check
154	that another request had started (500).
155	- `worker: exited (<how>)`: the worker has exited. `<how>` is how it
156	ended, such as `exit status 1` or `signal: killed`, or `its status pipe
157	closed` when the app saw it go before learning how (500).
158	- `worker: did not start: <error>`: the app could not start the worker
159	(500).
160	- `worker: no status for <n>s`: the worker has not reported a write for 3
161	seconds or more, for example because it is stuck (500).
162	- `worker: no row committed since start`: the worker has not written a row
163	since the app started, for example in its first moments or if it did not
164	start (500).
165	- `worker: no row committed since start; last error: <error>`: as above,
166	and its writes are failing (500).
167	- `worker: the last write failed: <error>`: the worker's latest write
168	failed (500).
169
170	On `/report`:
171
172	- The JSON report (200). Its fields are `host`, `region`, `region_from`,
173	`mode`, `boot_id`, `rows` (each with `kind`, `region`, `count`,
174	`min_seq`, `max_seq`), `fired` (each with `kind`, `name`, `due_at`,
175	`count`), `file_key_sha256`, `platform_key_sha256`, `object` (with
176	`key`, `body` and, when it could not be read, `error`), `has_secret`,
177	`worker_pid`, `worker_writes_committed`, `worker_writes_failed`,
178	`worker_failed_in_a_row`, `worker_last_error`, `worker_exit` and
179	`worker_last_status_at` (unix seconds, 0 before the first).
180	- `report: <error>`: a database read failed (500).
181
182	The values the app writes into the report itself:
183
184	- `region`: the value of `CARLOS_REGION`, or `home` when the box has no
185	region.
186	- `region_from`: `CARLOS_REGION` when the variable is set, and
187	`unset, treated as home` when it is not.
188	- `mode`: `normal`, or `after-write` when the failure switch names this
189	region with `,after-write`. Before-write mode sends no report.
190	- `kind` in `rows`: `bg` for the background loop, `worker` for the
191	worker, `tick` and `once` for timer fires. `kind` in `fired`: `tick` or
192	`once`.
193	- `object.key`: `probe/<boot id>.txt`, the object this boot wrote.
194	- `object.body`: the object's contents as read back. The app writes
195	`moveprobe boot <boot id> host <host> region <region> at <time>`,
196	followed by a new line, where the time is UTC in RFC 3339 form.
197	- `object.error`: `store: <names> not set` when the store variables are
198	missing, or `get <key>: <error>` when the object cannot be read back.
199	- `worker_last_error`: the database's own error text for the worker's
200	latest failed write. The app adds no words of its own.
201	- `worker_exit`: empty while the worker runs; otherwise
202	`exited (<how>)` or `did not start: <error>`, as on `/ready` without the
203	`worker: ` prefix.
204
205	On `/arm-once`:
206
207	- `{"name": "once-<n>", "at": "<time>", "due_at": <unix seconds>}`: the
208	timer is armed (200).
209	- `in: want whole seconds, at most 400 days`: `in` is missing or out of
210	range (400).
211	- `no control socket: CARLOS_CONTROL_SOCKET is not set`: the instance has
212	no control socket (503).
213	- `CARLOS_ADMIN_TOKEN is not set`: the instance has no admin token, so it
214	cannot arm timers (503).
215	- `database: <error>`: the timer number could not be taken (500).
216	- `control socket: <error>`: the request to the control socket could not be
217	made or sent (500 or 502).
218	- `control socket: <status> <reply>`: the control socket refused the timer
219	(502).
220
221	On `/tick` and `/once`:
222
223	- `ok`: the fire is recorded (200).
224	- `want X-Carlos-Schedule and X-Carlos-Schedule-At`: a fire without the
225	timer's name or due time (400).
226	- `database: <error>`: the row could not be written (500).
227
228	On `/report`, `/arm-once`, `/tick` and `/once`:
229
230	- `unauthorized`: the bearer is missing or wrong (401).
231
232	Anywhere else, Go's own answers: `404 page not found` for an unknown path,
233	and `Method Not Allowed` for a known path with the wrong method (405).
234