moveprobe

moveprobe is a small app for rehearsing instance moves. It keeps state in every place a real app can, so that after a move you can check that each kind of state arrived. It is not part of the platform release. You build it and ship it like any other app.

What it keeps

• Database rows. A background loop writes a row every half second. A separate worker process writes one every second. Each timer fire writes one. Every row records the region it was written in.
• A key file. On first boot it writes instance.key into the instance's files directory and records the same key in the database. Every boot checks that the two still match.
• The platform's instance key. On first boot it records a fingerprint of CARLOS_INSTANCE_KEY. Every boot checks that the key is still set and unchanged.
• An object in its store. On every boot it writes probe/<boot id>.txt to the store bound with the prefix MOVEPROBE, and reads it back on every readiness check.
• A sealed secret. It needs MOVEPROBE_SECRET to be set.
• Timers. A once-timer you arm through the app, and a schedule you set with carlos schedule set.

Build

The rehearsal boxes are arm64:

CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o moveprobe ./cmd/moveprobe

Deploy

Run these signed in to the ops account, unless the rehearsal names another. Use the deployment's instance domain where these say example.com, and pick a host name no earlier rehearsal has used.

carlos ship --app moveprobe --label "moveprobe" ./moveprobe
carlos instances enable --app moveprobe
carlos store create --app moveprobe --env-prefix MOVEPROBE
carlos secrets set --app moveprobe MOVEPROBE_SECRET=<a long random value>
carlos schedule set --app moveprobe --name every-minute --every 1m --path /tick
carlos instances create --app moveprobe --host mp1.example.com --region ie

Leave private storage off when you enable instances, and create the store without --environment. A move only carries the app's own store. It refuses an instance whose store belongs to an environment or to the instance itself. The account must be approved for object storage before store create provisions anything.

Keep the value of MOVEPROBE_SECRET in your shell. It is also the password for /arm-once and /report.

Use it

curl https://mp1.example.com/ready
curl -X POST -H "Authorization: Bearer $MOVEPROBE_SECRET" "https://mp1.example.com/arm-once?in=600"
curl -H "Authorization: Bearer $MOVEPROBE_SECRET" https://mp1.example.com/report

| path | who can call it | what it does | |---|---|---| | / and /ready | anyone | 200 when the key file, the platform key, the stored object and the secret all check out, and the worker's writes are landing. Otherwise 500 with every failed check. The store check runs at most once every 5 seconds, and requests in between share its result. A move probes /. | | GET /report | MOVEPROBE_SECRET | JSON: row counts by kind and region with their lowest and highest sequence numbers, the timer fires by name and due time, both key fingerprints, the stored object, whether the secret is set, the worker's pid and failed writes, and the region the app saw. Any failed database read makes it a 500, never a partial report. | | POST /arm-once?in=<seconds> | MOVEPROBE_SECRET | Arms a once-timer named once-<n> that calls /once after that many seconds. Answers 503 when the instance has no control socket. | | POST /tick, POST /once | the platform | Timer fires. Each one writes a row with the timer's name and due time. |

The background rows are numbered in one sequence that carries on across boots and regions. If no row is missing, the count of bg rows equals the highest bg sequence number.

The region comes from CARLOS_REGION. On a box that has no region, the app uses home, and /report says so.

Make readiness fail on purpose

MOVEPROBE_FAIL_READY_IN=<region> makes the app fail its readiness check in that region. The app reads it when it starts, so set it before you request the move, and remove it once the run's checks are recorded:

carlos env set --app moveprobe MOVEPROBE_FAIL_READY_IN=ie
carlos env unset --app moveprobe MOVEPROBE_FAIL_READY_IN

There are two modes:

• MOVEPROBE_FAIL_READY_IN=ie writes nothing at all in ie. It does not open the database, touch the files directory, start the worker or call the control socket. Every path answers 503. A rollback from here should be clean.
• MOVEPROBE_FAIL_READY_IN=ie,after-write writes one background row in ie, starts the worker, and then answers 503 on / and /ready. A rollback from here is dirty, and the source should end up holding that row.

In any other region the variable has no effect.

The worker

The app starts a worker process in its own session, with no connection to the app's output. Stopping the app does not stop the worker. That is the point: it shows whether a move's stop really stops every process the instance started. Once the app has gone, the worker keeps writing for 60 seconds and then exits by itself.

The worker tells the app whether each write landed. /report shows how many committed and how many failed since the app started, how many of the latest failed in a row, the last error, when it last heard from the worker, and why the worker is gone if it is. /ready fails until a worker row has committed since the app started, whenever the worker's latest write failed, if the worker has sent nothing for 3 seconds, and once the worker has exited.

Strings

Every body the app can send, and when. Text in angle brackets is filled in when the app answers. /ready joins its failed checks with ; .

On / and /ready:

• ready: every check passed (200).
• before-write mode: this box writes nothing: the failure switch names this region with no suffix. Every path gives this answer (503).
• after-write mode: not ready here: the failure switch names this region with ,after-write (503).
• key mismatch: instance.key exists but the database has no key: the files directory has a key file, but the database never recorded one (500).
• key mismatch: instance.key is missing: the database has a key, but the file is gone (500).
• key mismatch: instance.key differs from the database: the file and the database hold different keys (500).
• key mismatch: instance.key: <error>: the file is there but cannot be read (500).
• CARLOS_INSTANCE_KEY is not set: the platform key is missing (500).
• CARLOS_INSTANCE_KEY differs from the one seen at first boot: the platform key changed (500).
• MOVEPROBE_SECRET is not set: the sealed secret is missing (500).
• store: <names> not set: one or more of the store variables is missing (500).
• store: put <key>: <error>: the boot's object could not be written (500).
• store: get <key>: <error>: the boot's object could not be read back (500).
• store: <key> reads back different bytes: the object came back changed (500).
• store: <error>: the request gave up while waiting for a store check that another request had started (500).
• worker: exited (<how>): the worker has exited. <how> is how it ended, such as exit status 1 or signal: killed, or its status pipe closed when the app saw it go before learning how (500).
• worker: did not start: <error>: the app could not start the worker (500).
• worker: no status for <n>s: the worker has not reported a write for 3 seconds or more, for example because it is stuck (500).
• worker: no row committed since start: the worker has not written a row since the app started, for example in its first moments or if it did not start (500).
• worker: no row committed since start; last error: <error>: as above, and its writes are failing (500).
• worker: the last write failed: <error>: the worker's latest write failed (500).

On /report:

• The JSON report (200). Its fields are host, region, region_from, mode, boot_id, rows (each with kind, region, count, min_seq, max_seq), fired (each with kind, name, due_at, count), file_key_sha256, platform_key_sha256, object (with key, body and, when it could not be read, error), has_secret, worker_pid, worker_writes_committed, worker_writes_failed, worker_failed_in_a_row, worker_last_error, worker_exit and worker_last_status_at (unix seconds, 0 before the first).
• report: <error>: a database read failed (500).

The values the app writes into the report itself:

• region: the value of CARLOS_REGION, or home when the box has no region.
• region_from: CARLOS_REGION when the variable is set, and unset, treated as home when it is not.
• mode: normal, or after-write when the failure switch names this region with ,after-write. Before-write mode sends no report.
• kind in rows: bg for the background loop, worker for the worker, tick and once for timer fires. kind in fired: tick or once.
• object.key: probe/<boot id>.txt, the object this boot wrote.
• object.body: the object's contents as read back. The app writes moveprobe boot <boot id> host <host> region <region> at <time>, followed by a new line, where the time is UTC in RFC 3339 form.
• object.error: store: <names> not set when the store variables are missing, or get <key>: <error> when the object cannot be read back.
• worker_last_error: the database's own error text for the worker's latest failed write. The app adds no words of its own.
• worker_exit: empty while the worker runs; otherwise exited (<how>) or did not start: <error>, as on /ready without the worker: prefix.

On /arm-once:

• {"name": "once-<n>", "at": "<time>", "due_at": <unix seconds>}: the timer is armed (200).
• in: want whole seconds, at most 400 days: in is missing or out of range (400).
• no control socket: CARLOS_CONTROL_SOCKET is not set: the instance has no control socket (503).
• CARLOS_ADMIN_TOKEN is not set: the instance has no admin token, so it cannot arm timers (503).
• database: <error>: the timer number could not be taken (500).
• control socket: <error>: the request to the control socket could not be made or sent (500 or 502).
• control socket: <status> <reply>: the control socket refused the timer (502).

On /tick and /once:

• ok: the fire is recorded (200).
• want X-Carlos-Schedule and X-Carlos-Schedule-At: a fire without the timer's name or due time (400).
• database: <error>: the row could not be written (500).

On /report, /arm-once, /tick and /once:

• unauthorized: the bearer is missing or wrong (401).

Anywhere else, Go's own answers: 404 page not found for an unknown path, and Method Not Allowed for a known path with the wrong method (405).