eleven / messenger Public

bot: the promised-op stall — self-heal catches fetch-promises and can propose service_op (#522)

Live failure (Paul, first real "@eleven what are the open issues" ask): the
model replied "Great! Let me pull up the open issues for you." with NO tool
call — and nothing happened. Two holes in the safety net, both fixed:

- promisedButton only watched for tap/button/chip, so the fetch-promise
  family ("let me pull up / I'll fetch / one moment while I query") never
  triggered the forced retry. The pattern now catches promise-verbs + fetch
  verbs, pinned by a match/miss table including the exact live phrase. A
  model cannot fetch anything itself, so a promise with no tool call is
  always wrong; matching generously is cheap — the retry only ATTACHES a
  consent chip, nothing runs without the tap.
- The forced retry offered chipTools() only — it literally could not propose
  the op that was promised. retryTools() = chips + service_op (never the
  builders: a forced retry must not trigger a build or mint a bot).

Plus the system prompt now says it outright: never claim you'll fetch/pull
up/check anything — attach the service_op chip in that same reply; the
member's tap is the only way data moves.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Paul Campbell 4 months ago 8e6cc58568e3f87cd8bf82158b4a77198d4d96d5 parent 0d891df

2 files changed, +72 −5

internal/bot/brain.go +22 −5
internal/bot/brain_test.go +50 −0

diff --git a/internal/bot/brain.go b/internal/bot/brain.go

index 7ba3399..951af4c 100644

--- a/internal/bot/brain.go

+++ b/internal/bot/brain.go

@@ -129,7 +129,9 @@ func (r *Runtime) systemPrompt() string {

"(domains, code, payments) with the service_op tool — its description is the complete list of " +

"what each service can do. Writes always go through the member's confirmation tap; nothing runs " +

"from your proposal alone. If they haven't connected the service, the tap will tell them so — " +

- "proposing is still fine, or point them at Integrations (＋ New chat → Integrations) to connect."

+ "proposing is still fine, or point them at Integrations (＋ New chat → Integrations) to connect. " +

+ "You cannot fetch, pull up, check, or list anything from a service yourself — NEVER say you will; " +

+ "the only way data moves is the member tapping a service_op chip you attach in that same reply."

if r.appGuide != "" {

s += "\n\nYou CAN build small personal mini-apps for members — private tools like trackers, " +

"lists, counters, tiny games — via the create_app tool. When someone asks for an app (or " +

@@ -169,7 +171,7 @@ func (r *Runtime) systemPrompt() string {

// instance serves the authoring guide. Everything except create_app becomes a

// CONSENT CHIP — the client whitelists, and nothing runs until a tap.

func (r *Runtime) replyTools() []anthTool {

- tools := append(chipTools(), serviceOpTool())

+ tools := retryTools()

if r.appGuide != "" {

tools = append(tools, createAppTool())

}

@@ -179,6 +181,16 @@ func (r *Runtime) replyTools() []anthTool {

return tools

}

+// retryTools is the promised-button forced retry's palette: every CONSENT

+// CHIP the model may attach — the basic chips AND service_op (the live stall

+// this fixes: "let me pull up the open issues" retried into a palette that

+// couldn't propose the op). Never the builders: a forced retry must not be

+// able to trigger a build or mint a bot, but forcing a chip is safe — nothing

+// runs without the member's tap.

+func retryTools() []anthTool {

+ return append(chipTools(), serviceOpTool())

// serviceOpTool is the connected-services proposal tool. Its description IS

// the embedded registry — the closed list of ops the manifests declare is the

// closed list the model can propose, and actionsFromToolUse re-checks every

@@ -258,7 +270,13 @@ func (r *Runtime) buildContext(tc *ThreadCtx) ([]anthMessage, error) {

return out, nil

}

-var promisedButton = regexp.MustCompile(`(?i)\b(tap|button|chip)\b`)

+// promisedButton catches a reply that PROMISES an action instead of attaching

+// one: the classic "tap the button" with no tool call, and the fetch-promise

+// family ("Great! Let me pull up the open issues for you." — the live stall

+// that shipped) — the model cannot fetch anything itself, so a promise with

+// no tool call is always wrong. Matching generously is cheap insurance: the

+// forced retry only ATTACHES a consent chip; nothing runs without the tap.

// modelReply is the conversational path: build context, ask Claude (adaptive

// thinking, chip tools), self-heal the promised-button case, return the reply.

@@ -375,8 +393,7 @@ func (r *Runtime) modelReply(tc *ThreadCtx) (Reply, bool) {

// Forced tool_choice requires thinking off; omitting the field is

// "off" and every model accepts it (an explicit thinking type here

// would 400 on Haiku, which is exactly the bug this path must avoid).

- // Chip tools only: a forced retry must never trigger a build.

- Tools: chipTools(),

+ Tools: retryTools(),

ToolChoice: map[string]any{"type": "any"},

Messages: context,

})

diff --git a/internal/bot/brain_test.go b/internal/bot/brain_test.go

index 2f79d51..1493e5d 100644

--- a/internal/bot/brain_test.go

+++ b/internal/bot/brain_test.go

@@ -60,3 +60,53 @@ func TestAdaptiveThinking(t *testing.T) {

}

+// TestPromisedButtonCatchesFetchPromises pins the self-heal trigger: the

+// classic button-promises AND the fetch-promise family — the exact live stall

+// ("Great! Let me pull up the open issues for you.", no tool call, no chip,

+// nothing happens) must match, ordinary conversation must not.

+func TestPromisedButtonCatchesFetchPromises(t *testing.T) {

+ matches := []string{

+ "Tap the button below.",

+ "Here's a chip for that.",

+ "Great! Let me pull up the open issues for you.",

+ "I'll fetch the list now.",

+ "Sure — let me check the domains on your account.",

+ "One moment while I query the repository.",

+ "I will list them for you.",

+ }

+ for _, s := range matches {

+ if !promisedButton.MatchString(s) {

+ t.Errorf("should match (a promise with no tool call): %q", s)

+ }

+ misses := []string{

+ "Want help setting up a reminder for yourself instead?",

+ "Here you go — three domains, all healthy.",

+ "You can connect GitHub from Integrations.",

+ "That's a great idea!",

+ }

+ for _, s := range misses {

+ if promisedButton.MatchString(s) {

+ t.Errorf("should NOT match (ordinary conversation): %q", s)

+ }

+// TestRetryToolsIncludeServiceOp pins the forced retry's palette: it must be

+// able to propose the op that was promised (service_op — the live stall) and

+// must never include the builders (a forced retry cannot trigger a build).

+func TestRetryToolsIncludeServiceOp(t *testing.T) {

+ names := map[string]bool{}

+ for _, tool := range retryTools() {

+ names[tool.Name] = true

+ }

+ if !names["service_op"] {

+ t.Fatal("forced retry cannot propose service_op — the promised op stays unfulfillable")

+ }

+ for _, banned := range []string{"create_app", "create_bot", "remove_bot"} {

+ if names[banned] {

+ t.Fatalf("forced retry must never offer %s", banned)

+ }