Our support bot kept escalating questions it already knew the answer to

One message stopped testing cold: "my bowl is not getting connected to wifi." We have an FAQ that answers exactly this. The bot escalated anyway — no answer sent, just a ticket sitting in Slack for someone to pick up six hours later.

Nothing was broken, technically. Retrieval ran, found the FAQ, scored it, compared the score to our threshold, and the score lost. 0.595 against a cutoff of 0.6. I'd set that cutoff myself, a few days earlier, off a test file with sixteen hand-written questions in it. It felt scientific at the time — off-topic queries scored 0.53–0.54, real matches scored 0.60 and up, so 0.6 looked like a clean line. It just turned out to be a clean line drawn through a very small, very fake dataset, and a perfectly ordinary way of asking the question landed right on top of it.

The instinct is to go fix the number. Lower it to 0.55, rerun the sixteen questions, ship it. I almost did, until it was obvious that whatever number I picked, someone would eventually phrase a real question just underneath it, and I'd be back here in a month arguing about a different decimal. The actual problem wasn't the threshold. It was that the threshold was the only thing deciding whether the model got to speak at all.

What retrieval was actually doing

Before this, the flow was: embed the question, pull the top FAQs, check if the best score clears the bar. Clear it, and the LLM gets to generate an answer from that FAQ. Miss it, and the code never calls the LLM — it just escalates straight to Slack. Retrieval wasn't feeding the model information. It was deciding, on its own, whether the model was even allowed to try.

That's a strange amount of trust to put in a cosine similarity score. It tells you how close two pieces of text landed in embedding space. It doesn't know that "my bowl won't connect to wifi" and "the wifi setup for my bowl isn't working" are the same question asked two different ways by two different people typing on their phones. It also doesn't know the difference between "no relevant FAQ exists" and "the FAQ exists, but not many people have phrased it your way in training data." Below-threshold got treated as the first case every time, when a lot of the time it was actually the second.

So we stopped gating on it

The fix ended up being almost annoyingly simple to describe, harder to actually get right: call the LLM on every message, always, and hand it whatever retrieval found — even if that's a weak match, even if it's nothing. Let the model be the one to decide whether what it has is enough to actually answer with.

Which immediately raises the next question: if the model always answers, how do you tell "it actually used the FAQ" apart from "it politely made something up" or "it just said hi back"? You can't get that from retrieval anymore, because retrieval isn't gating the decision. So we made the model tell us directly.

Making the model grade its own homework

Every reply now comes back as a structured tool call with two fields — the answer text, and an intent: faq_answered if it actually grounded the reply in what retrieval gave it, small_talk if the message was a greeting or chatter with no real question in it.

RESPOND_TOOL = {
"name": "respond",
"input_schema": {
"type": "object",
"properties": {
"answer": {"type": "string"},
"intent": {
"type": "string",
"enum": ["faq_answered", "small_talk"],
},
},
"required": ["answer", "intent"],
},
}

One thing that took me a minute to get right: I originally reached for toolChoice: {"tool": {"name": "respond"}} to force it to always use this schema — seemed like the obvious move. Llama 4 Scout on Bedrock rejected it outright: This model doesn't support the toolConfig.toolChoice.tool field. Tried {"any": {}} too, same rejection. Turns out the only supported option is {"auto": {}} — the model decides for itself whether to call a tool. So the system prompt's instruction to "always call the respond tool" is doing real work, not backed by an API guarantee. It's been reliable across every test I've thrown at it, but I made sure the code treats a response that skips the tool call as a hard failure rather than assuming it'll never happen.

intent doesn't get to trigger escalation on its own, by the way — that's a separate check, a groundedness score from Bedrock Guardrails, running independently. Whether the model actually answered and whether a human needs to step in turned out to be two different questions that our old single threshold was quietly answering both at once. Splitting them is most of what made this work.

And splitting them is exactly what surfaced the next bug, which I did not see coming at all.

The bug where our own personalization broke us

We'd just added the bot using the asker's name in replies — "Hi Aarthi, here's how to..." — a nice small touch. Retesting the same wifi question after the retrieval fix, it failed again, in a completely different way. I pulled the raw guardrail trace expecting a grounding failure. Grounding scored 0.75, passing. Relevance scored 0.84, passing. The answer was, by every number that mattered, correctly grounded. And it still came back as blocked.

Buried in the same trace: the guardrail's PII policy had fired and redacted "Aarthi" as a detected name. Totally correct behavior for a PII filter. The problem was our code, which read the guardrail's top-level GUARDRAIL_INTERVENED flag and treated any intervention — from any policy, for any reason — as "this answer is ungrounded." One boolean, three unrelated policies feeding into it. Our own personalization feature had accidentally become a bug in our groundedness check.

for assessment in result.get("assessments", []):
filters = assessment.get("contextualGroundingPolicy", {}).get("filters", [])
if any(f.get("action") == "BLOCKED" for f in filters):
return False
return True

The fix was to stop reading the top-level flag and only look at what the contextual-grounding policy itself said. This function is shared with our existing vet-chatbot feature too, so the same bug had probably been quietly firing there as well — one fix, two features.

Guardrails needed the same scrutiny as everything else

A couple more things broke in ways I wouldn't have predicted from reading the config. Our off-topic filter was written as a negation — "requests unrelated to EverBowl setup/use/troubleshooting" — and it was blocking real questions that didn't happen to say the word "EverBowl" by name, because nobody actually says "EverBowl," they say "my bowl." Rewriting the same rule as a positive list of what's actually off-topic (homework help, coding requests, unrelated pet health questions, that kind of thing) fixed it without loosening anything real. Verified both directions after: the wifi question now answers correctly, and "write me a python script to scrape a website" still gets blocked, same as before.

Separately, a relevance filter meant to catch answers that pass grounding but don't really address the question kept scoring as low as 0.42 on answers that were obviously fine to a human reading them. I didn't try to re-tune it. Our retrieval threshold, the "say you're not sure" prompt instruction, and a phrase-matching safety net were already covering the case this filter was supposed to catch, and a flaky check that occasionally blocks good answers is worse than not having it — it just teaches you to distrust every block it produces, including the real ones. Dropped it, bumped the guardrail version, moved on.

The one that came from an actual phone, not a test script

Not every bug came from a trace. A tester messaged, half-joking, that the bot saying "Hi Aarthi" on nearly every reply was "kinda annoying." Which, fair. The instruction I'd written told the model to use the name "naturally once" — except each reply is generated fresh with no memory of the last one, so "once" reads to the model as "once per reply," not once per conversation. I reworded it to bias toward using the name rarely rather than routinely. Across four test replies afterward, it used the name once instead of four times — the initial greeting, and quiet after that.

Also found by a tester on an actual phone, not by me: they replied "I'm not sure" mid-conversation, just making conversation, and it got escalated. Turned out "I'm not sure" is literally one of the phrases we watch for as a sign the model is giving an honest refusal on a real question. The model had correctly classified the message as small_talk — it just happened to echo back a phrase that our own refusal-detector was watching for, and that detector ran before intent was even checked. Reordered the checks so small talk short-circuits immediately, before the refusal-phrase check ever runs. That check only means something when the model actually attempted a real answer.

What "escalated" should actually mean afterward

One thing I didn't want to guess at was what happens after a conversation escalates. While thinking this through I ended up watching an unrelated ElevenLabs voice-agent demo for a telecom company, just to see how someone else handled it — and even that demo dodged the question, with the agent saying something like "in a real scenario I'd stop answering here, but since this is a demo I'll keep going." That's the wrong answer for us, so I made it an explicit one instead of an implicit one: once a conversation is escalated, the bot never generates another reply, no matter how long the human takes to respond. One quiet, static, non-LLM-generated acknowledgment gets sent to the user the first time they follow up after escalating, so the conversation doesn't go completely silent — and never again after that, so it doesn't nag.

Where this actually leaves things

I'd be lying if I said this was all cleanly planned from the start. The threshold was a guess, the guess broke on real traffic, fixing it exposed a second bug in how we read guardrail output, and fixing that surfaced a naming collision between two features that had no idea the other existed. Most of the actual debugging here was less "clever architecture" and more "pull the raw trace, read every field in it, don't assume the summary boolean means what it says."

The thing I'd actually tell someone building one of these: don't let one number decide two different questions. "Is there relevant context" and "did the model use it" and "is a human needed" turned out to be three separate things worth checking separately, not one score doing triple duty. None of it is visible to a user — nobody's going to see an intent field. What they'll notice, hopefully, is that the bot answers the question they actually typed, wifi typo and all, instead of quietly giving up because a decimal came in a hair short.