eleven / messenger Public

tools+web+docs: multi-contributor prompt registry — per-contributor prefixes + manifest merge (#799)

Each machine publishes under prompts/<slug>/ (slug = kebab-cased
--contributor), so corpora never clobber each other; the tool merges this
machine's entry into the LIVE prompts/contributors.json (public read, no
creds) so uploading the manifest never drops other contributors, and warns
when built offline. The page fetches every listed contributor's
index/prompts/issues and merges client-side (chips/search/totals already
handled multiple contributors); flat pre-manifest corpora still load via
fallback. Cache reads fall back to the flat layout so an existing corpus's
Haiku summaries/gists survive the migration. docs/status.md § The prompt
registry documents how a contributor joins — with scoped bucket creds or by
handing a tarball to an operator.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Paul Campbell 3 months ago 1f6fe755f3247a96bb2eb5575a4b04c0bd5074cb parent 38935e6

3 files changed, +141 −25

docs/status.md +38 −5
tools/prompt-registry.mjs +58 −12
web/status/prompts.html +45 −8

diff --git a/docs/status.md b/docs/status.md

index 1940e08..41d109d 100644

--- a/docs/status.md

+++ b/docs/status.md

@@ -363,6 +363,37 @@ transcript, but that becomes a *requirement* on other contributors only once

agreed with them — don't encode it as a rule anywhere until that conversation

has happened.

+**Bucket layout is per-contributor** — corpora can never clobber each other:

+```

+prompts/contributors.json ← the manifest: [{ slug, name }] the page merges

+prompts/<slug>/index.json ← one prefix per contributor (slug = kebab-cased

+prompts/<slug>/prompts.json --contributor name); everything the tool

+prompts/<slug>/issues.json builds lands under the contributor's own slug

+prompts/<slug>/<session>.jsonl

+prompts/<slug>/<session>.sum.json

+```

+The tool merges this machine's entry into the *live* manifest (a public read —

+no creds needed) before writing contributors.json, so uploading it never drops

+other contributors. If it prints the OFFLINE warning, don't upload the

+manifest from that build — re-run online.

+**How another contributor joins** (their choice, per the policy above): run

+the tool on their machine with `--contributor "Their Name"` (plus their own

+`--personal` / `--sessions` files and the security sweep — same rules for

+everyone), then publish their prefix one of two ways:

+- **With bucket creds** (mint a scoped IAM key allowing PutObject on

+ `prompts/<their-slug>/*` + `prompts/contributors.json` only): run the two

+ `aws s3` commands the tool prints — `sync` of their own prefix, `cp` of the

+ manifest.

+- **Without creds**: `tar czf corpus.tgz -C <out>/prompts <slug>

+ contributors.json` and hand the tarball to an operator, who extracts and

+ runs the same two commands from the router box (`set -a; . /data/status.env;

+ set +a` for the creds). The operator never edits the corpus — the

+ contributor's build is what publishes, fail-closed scan and all.

Accruing a machine's sessions:

1. `node tools/prompt-registry.mjs --out <dir>` (from the repo root) — sweeps

@@ -373,11 +404,13 @@ Accruing a machine's sessions:

whole-corpus-per-machine for now; per-session accrual can come later if

two machines ever need to merge into one index.

2. Upload the corpus straight to the status bucket (it's far too large to

- commit — a quarter GB and growing):

- `aws s3 sync <dir>/prompts s3://eleven-status-<account>/prompts

- --cache-control max-age=300` (run it on the box with the status-publisher

- creds — `set -a; . /data/status.env; set +a` — or anywhere with

- equivalent PutObject rights).

+ commit — a quarter GB and growing). The tool prints the exact commands:

+ `aws s3 sync <dir>/prompts/<slug> s3://eleven-status-<account>/prompts/<slug>

+ --delete --cache-control max-age=300`, then `aws s3 cp` the manifest (run

+ on the box with the status-publisher creds — `set -a; . /data/status.env;

+ set +a` — or anywhere with equivalent PutObject rights). Sync **only your

+ own prefix**; the manifest upload is a `cp`, never a sync with `--delete`

+ at the `prompts/` root, which would erase other contributors.

3. `prompts.html` itself is a committed viewer page: copy it to the box's

`/opt/lchat/status-pages/` like the other pages; the 5-minute publish

re-uploads it.

diff --git a/tools/prompt-registry.mjs b/tools/prompt-registry.mjs

index 66e2d45..f3f6932 100644

--- a/tools/prompt-registry.mjs

+++ b/tools/prompt-registry.mjs

@@ -106,6 +106,10 @@ function sourceLabel() {

}

const source = sourceLabel();

const contributor = argAll("--contributor")[0] || source;

+// Multi-contributor layout: each machine publishes under its own slug prefix

+// (prompts/<slug>/…) so corpora never clobber each other; the page merges

+// every contributor listed in prompts/contributors.json.

+const slug = contributor.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "") || source;

// Session-level controls: withhold (tombstone) / omit (aggregate count only).

const sessionCtl = new Map();

@@ -279,7 +283,13 @@ console.log("transcript dirs: " + dirs.length + ", sessions: " + files.length +

const tmp = outRoot + ".building";

rmSync(tmp, { recursive: true, force: true });

-mkdirSync(join(tmp, "prompts"), { recursive: true });

+mkdirSync(join(tmp, "prompts", slug), { recursive: true });

+// Cache reads prefer the slug prefix, falling back to the pre-multi flat

+// layout so an existing corpus's summaries/gists survive the migration.

+const priorPath = (f) => {

+ const p = join(outRoot, "prompts", slug, f);

+ return existsSync(p) ? p : join(outRoot, "prompts", f);

+};

// Real user prompts in a user turn: plain-string content, or its text blocks —

// skipping tool results, harness wrappers (<system-reminder>, <command-…>),

@@ -342,7 +352,7 @@ for (const file of files) {

continue;

}

const body = outLines.join("\n") + "\n";

- if (!checkOnly) writeFileSync(join(tmp, "prompts", id + ".jsonl"), body);

+ if (!checkOnly) writeFileSync(join(tmp, "prompts", slug, id + ".jsonl"), body);

const redactions = Object.values(tally).reduce((a, b) => a + b, 0);

for (const [k, v] of Object.entries(tally)) grandTally[k] = (grandTally[k] || 0) + v;

sessions.push({ id, source, contributor, models: [...models].sort(),

@@ -360,7 +370,7 @@ const HAIKU = "claude-haiku-4-5-20251001";

const apiKey = process.env.ANTHROPIC_API_KEY;

const prevSummaries = new Map();

try {

- for (const s of JSON.parse(readFileSync(join(outRoot, "prompts", "index.json"), "utf8")).sessions) {

+ for (const s of JSON.parse(readFileSync(priorPath("index.json"), "utf8")).sessions) {

if (s.summary) prevSummaries.set(s.id + ":" + s.lines, s.summary);

}

} catch {} // first run, or no prior corpus — nothing cached

@@ -440,7 +450,7 @@ if (apiKey && !checkOnly) {

// rebuild never loses them. Transcripts are append-only, so keys never drift.

if (!checkOnly) {

const loadPrior = (id) => {

- try { return JSON.parse(readFileSync(join(outRoot, "prompts", id + ".sum.json"), "utf8")).gists || {}; }

+ try { return JSON.parse(readFileSync(priorPath(id + ".sum.json"), "utf8")).gists || {}; }

catch { return {}; }

};

let calls = 0, failed = 0, total = 0, cachedN = 0;

@@ -463,7 +473,7 @@ if (!checkOnly) {

const g = redactString(gists[r.k], {}); // re-pass today's rules

if (g) keep[r.k] = g;

}

- if (Object.keys(keep).length) writeFileSync(join(tmp, "prompts", id + ".sum.json"), JSON.stringify({ gists: keep }));

+ if (Object.keys(keep).length) writeFileSync(join(tmp, "prompts", slug, id + ".sum.json"), JSON.stringify({ gists: keep }));

}

}));

console.log(apiKey

@@ -486,7 +496,7 @@ if (!checkOnly) {

}

let issues = {};

- try { issues = JSON.parse(readFileSync(join(outRoot, "prompts", "issues.json"), "utf8")); } catch {}

+ try { issues = JSON.parse(readFileSync(priorPath("issues.json"), "utf8")); } catch {}

let fetched = 0, failedI = 0;

for (const u of urls) {

if (issues[u]) continue;

@@ -500,7 +510,7 @@ if (!checkOnly) {

}

const keepI = {};

for (const u of urls) if (issues[u]) keepI[u] = issues[u];

- writeFileSync(join(tmp, "prompts", "issues.json"), JSON.stringify(keepI));

+ writeFileSync(join(tmp, "prompts", slug, "issues.json"), JSON.stringify(keepI));

console.log("issue titles: " + Object.keys(keepI).length + "/" + urls.size + " resolved (" + fetched + " fetched, " + failedI + " failed)");

}

@@ -515,16 +525,45 @@ const index = {

sessions,

};

if (!checkOnly) {

- writeFileSync(join(tmp, "prompts", "index.json"), JSON.stringify(index, null, 1));

- writeFileSync(join(tmp, "prompts", "prompts.json"),

+ writeFileSync(join(tmp, "prompts", slug, "index.json"), JSON.stringify(index, null, 1));

+ writeFileSync(join(tmp, "prompts", slug, "prompts.json"),

JSON.stringify({ generated_at: index.generated_at, prompts: allPrompts }));

}

+// ---- contributors manifest ----------------------------------------------------

+// Merge this machine's entry into the LIVE manifest (public read, no creds

+// needed) so uploading it never drops other contributors. If the manifest

+// can't be reached, only this machine's entry is written and the log says

+// not to upload contributors.json from that build.

+const MANIFEST_URL = process.env.LCHAT_REGISTRY_MANIFEST_URL ||

+ "https://status.elevenmessenger.com/prompts/contributors.json";

+let manifestLive = false;

+if (!checkOnly) {

+ let manifest = { contributors: [] };

+ try {

+ const r = await fetch(MANIFEST_URL);

+ if (r.ok) {

+ const j = await r.json();

+ if (Array.isArray(j.contributors)) { manifest = j; manifestLive = true; }

+ } else if (r.status === 403 || r.status === 404) {

+ manifestLive = true; // the bucket answered: no manifest yet — first publish

+ }

+ } catch {}

+ manifest.contributors = [...manifest.contributors.filter((c) => c.slug !== slug), { slug, name: contributor }]

+ .sort((a, b) => a.slug.localeCompare(b.slug));

+ writeFileSync(join(tmp, "prompts", "contributors.json"), JSON.stringify(manifest, null, 1));

// ---- fail-closed verification --------------------------------------------------

let bad = 0;

if (!checkOnly) {

- for (const f of readdirSync(join(tmp, "prompts"))) {

- const hits = detect(readFileSync(join(tmp, "prompts", f), "utf8"), f.endsWith(".json"));

+ const files = [

+ ...readdirSync(join(tmp, "prompts")).filter((f) => f.endsWith(".json")).map((f) => join(tmp, "prompts", f)),

+ ...readdirSync(join(tmp, "prompts", slug)).map((f) => join(tmp, "prompts", slug, f)),

+ ];

+ for (const path of files) {

+ const f = basename(path);

+ const hits = detect(readFileSync(path, "utf8"), f.endsWith(".json"));

if (hits.length) { bad++; console.error("LEAK " + f + ": " + hits.join(", ")); }

}

if (bad) {

@@ -541,4 +580,11 @@ const mb = (n) => (n / 1048576).toFixed(1) + " MB";

console.log("sessions: " + sessions.length + ", prompts: " + index.totals.prompts +

", redactions: " + index.totals.redactions + " " + JSON.stringify(grandTally));

console.log("corpus: " + mb(sessions.reduce((a, s) => a + (s.bytes || 0), 0)) +

- (checkOnly ? " (check-only, nothing written)" : " → " + join(outRoot, "prompts")));

+ (checkOnly ? " (check-only, nothing written)" : " → " + join(outRoot, "prompts", slug)));

+if (!checkOnly) {

+ console.log("\nsync (own prefix only, then the manifest):" +

+ "\n aws s3 sync " + join(outRoot, "prompts", slug) + " s3://<status-bucket>/prompts/" + slug + " --delete --cache-control max-age=300" +

+ (manifestLive

+ ? "\n aws s3 cp " + join(outRoot, "prompts", "contributors.json") + " s3://<status-bucket>/prompts/contributors.json --cache-control max-age=300"

+ : "\n ⚠ manifest built OFFLINE — do NOT upload contributors.json (it would drop other contributors); re-run online first"));

diff --git a/web/status/prompts.html b/web/status/prompts.html

index 843ded2..b9363fd 100644

--- a/web/status/prompts.html

+++ b/web/status/prompts.html

@@ -280,18 +280,55 @@

return d;

}

+ // The corpus is per-contributor: prompts/contributors.json lists every

+ // slug, each with its own index/prompts/issues under prompts/<slug>/…, and

+ // the page merges them here. A corpus published before the manifest existed

+ // (flat prompts/index.json) still loads via the fallback.

+ const prefixById = new Map(); // session id → "slug/" for transcript fetches

+ const jf = async (p) => { const r = await fetch(p); if (!r.ok) throw 0; return r.json(); };

let index, promptsDoc;

try {

- index = await (await fetch("./prompts/index.json")).json();

- promptsDoc = await (await fetch("./prompts/prompts.json")).json();

+ let sessions = [], prompts = [], why = {}, omitted = 0, source = null;

+ try {

+ const manifest = await jf("./prompts/contributors.json");

+ const parts = (await Promise.all((manifest.contributors || []).map(async (c) => {

+ try {

+ const idx = await jf("./prompts/" + encodeURIComponent(c.slug) + "/index.json");

+ const pr = await jf("./prompts/" + encodeURIComponent(c.slug) + "/prompts.json").catch(() => ({ prompts: [] }));

+ const iss = await jf("./prompts/" + encodeURIComponent(c.slug) + "/issues.json").catch(() => ({}));

+ return { slug: c.slug, idx, pr, iss };

+ } catch { return null; } // listed but not uploaded yet

+ }))).filter(Boolean);

+ if (!parts.length) throw 0;

+ for (const p of parts) {

+ for (const s of p.idx.sessions || []) { prefixById.set(s.id, p.slug + "/"); sessions.push(s); }

+ prompts.push(...(p.pr.prompts || []));

+ Object.assign(why, p.idx.why || {});

+ Object.assign(ISSUES, p.iss || {});

+ omitted += (p.idx.totals && p.idx.totals.omitted) || 0;

+ source = source || p.idx.source;

+ }

+ } catch {

+ // pre-manifest flat layout

+ const idx = await jf("./prompts/index.json");

+ sessions = idx.sessions || []; why = idx.why || {};

+ omitted = (idx.totals && idx.totals.omitted) || 0; source = idx.source;

+ prompts = (await jf("./prompts/prompts.json").catch(() => ({ prompts: [] }))).prompts || [];

+ try { Object.assign(ISSUES, await jf("./prompts/issues.json")); } catch {}

+ }

+ sessions.sort((a, b) => (b.started_at || "").localeCompare(a.started_at || ""));

+ index = { sessions, why, source, totals: {

+ sessions: sessions.length,

+ prompts: sessions.reduce((a, s) => a + (s.prompts || 0), 0),

+ redactions: sessions.reduce((a, s) => a + (s.redactions || 0), 0),

+ withheld: sessions.filter((s) => s.withheld).length,

+ omitted,

+ } };

+ promptsDoc = { prompts };

} catch {

$("totals").textContent = "The registry corpus hasn’t been uploaded yet.";

return;

}

- try {

- const r = await fetch("./prompts/issues.json");

- if (r.ok) ISSUES = await r.json();

- } catch {} // no titles — GitHub links fall back to repo#n

WHY = index.why || {};

const allPrompts = promptsDoc.prompts || [];

const contributors = [...new Set(allPrompts.map((p) => p.c).filter(Boolean))];

@@ -482,13 +519,13 @@

$("viewer-status").textContent = "Loading transcript (" + fmtBytes(s.bytes) + ")…";

let raw, gists = {};

try {

- raw = await (await fetch("./prompts/" + encodeURIComponent(id) + ".jsonl")).text();

+ raw = await (await fetch("./prompts/" + (prefixById.get(id) || "") + encodeURIComponent(id) + ".jsonl")).text();

} catch {

$("viewer-status").textContent = "Couldn’t load the transcript.";

return;

}

try {

- const r = await fetch("./prompts/" + encodeURIComponent(id) + ".sum.json");

+ const r = await fetch("./prompts/" + (prefixById.get(id) || "") + encodeURIComponent(id) + ".sum.json");

if (r.ok) gists = (await r.json()).gists || {};

} catch {} // no gists — labels fall back to each reply's first line

$("viewer-status").hidden = true;