The trouble is that a meaningful share of real calls are not the easy call. And the failures are not evenly distributed. They cluster around five specific kinds of caller, and once you know what they are, you can test for them in about twenty minutes.

We run these tests on our own agents. We are publishing them because you should run them on ours too, and on everyone else's.

1. The interrupter

What happens: The agent starts listing options. The caller already knows what they want and cuts in three words into the list. The agent keeps talking. The caller talks louder. Both of them are now speaking, and neither is listening.

Why it breaks: Most deployed phone systems are half duplex, meaning they alternate strictly between listening and speaking. They wait for voice activity detection to decide the caller has finished, then respond. Interruption is not something the system handles badly so much as something it is not built to perceive at all.

This is the hardest problem in the field right now, not a minor polish issue. The 2026 ICASSP Human-like Spoken Dialogue Systems Challenge built a whole benchmark around it, releasing a dual-channel dataset of real human conversations specifically to measure how systems handle interruptions, overlapping speech and dynamic turn negotiation (HumDial Challenge, 2026).

How to test it: Call, ask a question that produces a long answer, and cut in after three or four words. Time how long the agent keeps talking. Under half a second and you have a system that models overlap. Two seconds and you have a phone tree with better vocabulary.

2. The caller on a bad line

What happens: Someone calls from a car with the window down, or a construction site, or a phone held at an angle, speaking English with an accent the model has not heard much of. The agent mishears the name, confirms the wrong appointment time, or asks the caller to repeat themselves three times until they give up.

Why it breaks: Two things compound here. Phone audio is band limited, carrying a much narrower slice of frequency than the studio-quality recordings most speech models are demonstrated on, so recognition accuracy on a real phone call is meaningfully worse than the number in any vendor's marketing.

On top of that, speech recognition accuracy is not uniform across speakers. A 2024 study in JAMIA Open evaluating transcription of patient speech in home healthcare found measurable disparities in system performance, consistent with earlier findings that commercial speech recognition produced error rates for African American speakers roughly double those for white speakers, with similar degradation documented for other dialects and accents underrepresented in training data (Zolnoori et al., JAMIA Open 2024).

If your customer base is not a demographic match for the average speech dataset, this is not an abstract fairness concern. It is a booking accuracy problem that shows up in your calendar.

How to test it: Call from a moving car with background noise. Better, have three people with different accents call and each spell out a name and a phone number. Check what actually landed in your system. Names and numbers are where this fails, and they are the two things that have to be right.

3. The caller who changes their mind

What happens: "I need Tuesday at three. Actually no, Wednesday. Wednesday morning if you have it, otherwise Tuesday is fine." The agent books Tuesday at three.

Why it breaks: People do not speak in finished sentences. Linguists call this repair, and it is constant in ordinary conversation: we start, revise, and correct ourselves mid-utterance without noticing. A system that treats the first clearly stated intent as the answer will get these calls wrong in a way that is worse than useless, because the caller believes the right thing was booked.

This is the failure that generates the angry call in week two.

How to test it: Book an appointment, then change the date halfway through the sentence. Then change it back. Then add a condition. See what ends up in the calendar, not what the agent says on the call, because those are frequently different.

4. The caller who wants something you do not offer

What happens: "Do you validate parking?" "Do you take Medicaid?" "Can I bring my dog?" Nothing in the agent's knowledge covers it. Then one of three things happens.

It guesses, which is the genuinely dangerous outcome. A confident wrong answer about insurance coverage or pricing creates a problem a missed call never would, and the caller will hold you to it.

It dead-ends, repeating that it cannot help until the caller hangs up.

Or it recognizes the boundary, says plainly that it does not have that information, and offers something useful instead. That third behavior is the whole ballgame, and it is a design decision rather than an emergent property of a good model.

How to test it: Ask three things that are plausible for your business but absent from your website. Parking, payment plans, whether someone specific is in today. Listen for whether the answer is grounded or invented. Then check what the system did next, because recognizing a gap and doing nothing about it is only half a solution.

5. The angry caller

What happens: Someone calls because something already went wrong. They are not looking for information, they are looking for acknowledgment and a person. The agent, cheerful and undeterred, offers to schedule an appointment.

Why it breaks: Most systems have no concept of emotional state, so escalation is triggered by keywords or by the caller explicitly asking for a human. But upset callers do not follow that script. They repeat themselves, they talk over the agent, they escalate gradually. By the time someone says "let me speak to a manager," the damage is done.

There is also a related finding worth knowing. In a field experiment published in Marketing Science, disclosing to customers that they were speaking with a bot before the conversation began cut purchase rates by roughly 80 percent and dramatically shortened calls, with customers rating the identical system as less knowledgeable and less empathetic once they knew (Luo et al., Marketing Science 2019). That study looked at outbound sales rather than inbound support, and attitudes have shifted since, but the mechanism is instructive: perceived empathy drops the moment people know. Which means an AI handling an upset caller starts from behind and needs to hand off faster than you would expect.

How to test it: Call and act frustrated. Interrupt. Repeat yourself. Do not ask for a human. Count how many turns it takes for the system to stop trying to resolve and start trying to transfer. This test is more uncomfortable than it sounds and it is the most informative one on the list.

Scoring it

Run all five against any system you are considering, including ours. Give each a pass or a fail, and write down what actually happened rather than how it felt.

Two failures out of five is normal for the category right now. Five out of five passing means either an unusually good system or an unusually easy test, so make the tests harder. And a system that passes the first four but fails the fifth is still a system you can deploy, as long as you know that upset callers need to reach a person quickly and you build for that rather than hoping it does not come up.

That is the honest framing. An AI receptionist is not a replacement for your front desk. It is coverage for the hours nobody was there, and relief from the routine calls that stop your team from doing anything else. Knowing precisely where it breaks is what lets you deploy it without embarrassment.

If you want to run these five against ours, start a free trial at aiassistant.co.