Caller Frustration Detection: Escalate Before They Hang Up

The patient has already decided to give up. Your AI receptionist has not noticed, because it is still waiting for a word she is too polite to use. In a peer-reviewed benchmark of a deployed receptionist and booking system, the keyword-based frustration detector was right every single time it fired, 100% precision, and it fired on 1% of the conversations that were actually frustrated.

That gap is the whole problem. Frustration shows up in the shape of a conversation long before it shows up in a voice: the same request asked twice, a task attempted and failed again, a call for a 30-second job running past 3 minutes. Those things are countable. Anger in a voice is not, at least not reliably, and not across every patient who calls you.

This post is about the trigger, not the handoff. It covers which signals are worth counting, where to set the threshold and why it moves, why acoustic sentiment loses to boring alternatives, how a speech-feature model fails some patients more than others, and the three cases that skip scoring entirely.

Most systems escalate on the wrong signal

Most deployments fire on one of two triggers: a magic word like agent or representative, or a low recognition confidence score. Both arrive late. A frustrated patient who never swears and speaks clearly trips neither, so the call runs on until she hangs up and books somewhere else.

A 2025 industry-track study measured this on a production system whose two evaluation domains were a receptionist transferring calls and a booking agent negotiating appointment slots. Its deployed detector matched profanity and negative words. Precision came out at 100%, recall at 1%, and the overall score at 0.41 Macro-F1. The authors' explanation is the line worth pinning up: poor conversation handling does not always show up as overtly negative language.

Confidence scores fail in the other direction. A low score means the system did not catch the words, which happens constantly with surnames and dates of birth. It says nothing about whether the caller minds. Plenty of calls recover from 3 bad transcriptions, and plenty of clean ones go badly because the answer was wrong.

The signals that actually predict a caller giving up

Frustration is a property of the conversation before it is a property of the voice. The signals that predict it are things you can count in a transcript: the same request asked twice, a task attempted twice, a rising turn count, a call running past the time its task should take.

Repetition earns the top spot. In that study's real deployment data, repeated utterances ran about 8% by fuzzy match and 9.6% by cosine similarity, against 2.1% and 4% in a public reference corpus of ordinary task dialogues. Repeating yourself is not normal conversation. It is what people do when the last answer did not land.

The feature set that beat every off-the-shelf sentiment and emotion model in the same benchmark was deliberately dull: semantic and syntactic repetition, how long each turn ran, how long the whole conversation ran, and how many turns it took. No emotion model, no acoustic features, and a Macro-F1 of 0.71 against 0.65 for the best sentiment model.

Interruptions and barge-ins belong in the log, but not in the trigger on their own. Natural turn-taking gaps run under 300 ms in ordinary speech, so a system listening for interruptions also hears every "mm-hm" as one. Track your false stop rate before you treat a barge-in as evidence of anything.

When to escalate on a weak signal

There is no single correct threshold, because the right one depends on what a miss costs. When missing a frustrated caller means losing a new patient or overlooking a clinical concern dressed up as annoyance, escalate on thin evidence. When the cost is 2 minutes of staff time, wait for something solid.

When a miss is expensive, a weak signal is enough. When it is cheap, wait for something solid.

The high-cost side covers more calls than most practices assume: anyone who has not booked yet, anyone calling a second time the same day, anyone whose reason for calling touches how they feel, and anyone raising a complaint. On those calls, 1 repeat is enough. You are not deciding whether the patient is angry. You are deciding whether a person should take it from here.

The low-cost side is the routine end: a known patient confirming a known appointment, an hours-and-parking question, a refill message. There, an unnecessary transfer costs a staff interruption and gains nothing, so wait for a second failed attempt or an explicit request.

Price both errors before you set a number. A false positive costs a few minutes at the front desk. A false negative costs you the patient, and no report shows it, because the call simply ended. If you have not decided which mistake you would rather make, you have accepted a vendor default rather than set a threshold.

Sentiment analysis is oversold

Vendors sell acoustic sentiment as though hearing emotion in a voice were a solved problem. It is not. In a published benchmark that pits these approaches against each other on real deployed traffic, off-the-shelf sentiment and emotion models lost to a feature set that counted repetition and turn structure alone.

The numbers: sentiment models scored 0.34 and 0.65 Macro-F1, emotion-detection models landed between 0.40 and 0.60, and the plain dialog-breakdown features scored 0.71. Large language models reading the conversation history did best at 0.83 to 0.87, which the authors report as a 33% relative improvement over the sentiment and emotion models and 22% over dialog breakdown.

The reason is not that emotion models are badly built. It is that they do not travel. Cross-corpus evaluations of speech emotion recognition report significant drops when a model trained on one corpus meets another, and one study concludes that the transfer of information across corpora is simply not successful without extra adaptation. Your phone line is another corpus: different handsets, different rooms, different people.

A speech-feature model does not fail evenly across your patients

Any trigger built on speech features inherits the accuracy gaps of the speech stack under it, and those gaps are documented and large. A model that mishears a patient more often will also read her turns as noisier, escalate her later, and leave no trace that it happened.

The clearest evidence is still Koenecke and colleagues in PNAS. Across 5 commercial systems from Amazon, Apple, Google, IBM and Microsoft, the systems misunderstood 35% of words spoken by Black speakers against 19% of words spoken by white speakers, and over 20% of Black speakers' samples were effectively unusable against under 2% of white speakers' samples. Same phrases, matched for age and gender.

Age moves it too. A speech model tested on interview audio from long-term care started at 48.3% word error rate and reached 24.3% after fine-tuning on that population, with residents the worst-served group at 22.7%. The authors name age, gender and accent as factors that change speech features and raise error rates. An 82-year-old speaking slowly is not a frustrated caller, and a trigger built on speech rate will read her as one. The wider set of design choices for those callers sits in what an AI receptionist does and where it stops.

Accent adds another layer. A 2025 study of synthetic voice services found technical performance disparities across 5 regional English accents and framed the result as a digital-exclusion risk.

The calls that escalate with no score at all

Three cases end the automated part of a call the moment they appear, and none of them is scored. Any clinical or urgent content. Any named symptom. Any request for a person. Your counters run after these, never before, so no frustration threshold can ever delay one of them.

Three cases end the automated call on the turn they appear. Nothing weighs them.

The symptom rule is the one people try to soften, usually by asking the system to judge whether a symptom sounds urgent enough. That judgement is triage, and an AI receptionist routes calls, it does not diagnose. A caller who mentions chest tightness while booking a cleaning gets a person, and nothing weighs how calm she sounded saying it.

A request for a person is a decision the caller already made. The mechanics of what happens next, warm or cold transfer, what data travels with her, what happens when nobody picks up, are covered in designing the handoff to a human. This post only sets the moment it fires.

Order these ahead of your counters in the configuration, not alongside them. If the no-score class is one branch among several, a scoring bug can swallow it. If it runs first and exits, it cannot.

Check what your trigger missed, then move it

A threshold you never audit is a guess with a timestamp on it. Once a month, sample calls from both sides: the ones where escalation fired, and the ones where it did not. The second sample is the one nobody pulls, and it is exactly where the expensive mistakes live.

Sample both sides. A trigger you never audit is a guess with a timestamp.

50 calls on each side is enough to see a pattern. Read them and mark each one 3 ways: a person should have taken this, a person should not have, or it is genuinely unclear. The unclear pile earns its keep, because it shows where your staff disagree about the rule.

The number that matters is the false negative: calls that ended without a human where one was needed. No report surfaces it, because a call that quietly ended looks identical to a call that went fine. You find these by reading them, or you do not find them.

Change one threshold per cycle. If you move the repeat count and the duration cap in the same month, you have learned nothing about either. Tolerance for this is not unlimited on the patient's side either, and the surveys behind whether patients accept an AI receptionist point the same way: acceptance tracks whether the task went well, not the technology behind it.

Field note

Before the trigger goes live

• The no-score cases run ahead of every counter: clinical content, a named symptom, a request for a person.
• Each counted signal is written to the call record, whether or not it fired.
• One named person owns the monthly sample of calls where escalation did not fire.
• A signed BAA covers the vendor storing your transcripts and recordings.

Set the trigger, then let the calls move it

Start narrow. Two counted signals, a second repeat and a second failed attempt at the same task, plus a duration cap for that task, with the no-score class sitting above all of them. That configuration is defensible on day 1, and it gives you something specific to argue with next month.

Order matters. The no-score class runs first, then the counters, in that sequence.

Walk the ladder once. Clinical content, a named symptom or a request for a person leaves on the turn it appears. A repeated request leaves on the second repeat. A task that has failed twice leaves with both attempts attached. A call past its time cap leaves on duration alone. Everything else keeps going, with the turn count written down.

Leave acoustic sentiment out of the trigger for now. Log its score beside your counted signals, compare the two against calls your staff labelled by hand, and promote it only if it beats counting on your own recordings. On the published evidence, it does not.

If you would rather size the problem before touching any configuration, the free Growth Leak Audit works out what your phone line is losing from your own numbers.