A contractor is calling you from a pickup on Highway 401. Window cracked. Indicator clicking. Phone on speaker somewhere near the cup holder. He says he needs to move tomorrow’s job, gives you a postal code, then spells a last name you’ve never heard before.
Or it’s a caller in rural Saskatchewan with one bar of signal, trying to confirm an appointment and reading out a 10-digit callback number before the line breaks up.
That’s the real test.
Not whether an AI voice agent sounded polished in a quiet demo. Whether it can understand an actual person on an actual Canadian phone line when the audio is messy, the accent is unfamiliar, and the call isn’t happening in a studio.
The demo score is not the driveway score
Most vendors lead with the cleanest number they have. In 2026, the best speech-to-text models can reach 95% to 98% word accuracy on clean audio. That figure is real, and it matters. It’s also the number least likely to match your incoming calls.
The spread across real scenarios is wide. AssemblyAI's 2026 accuracy breakdown lays it out plainly.
If you’re evaluating how an AI receptionist holds up on a bad phone line, that table should reset your expectations. A vendor quoting the clean-audio number is telling you something true but incomplete.
The industry metric behind these claims is Word Error Rate, or WER: (substitutions + insertions + deletions) / total words x 100. Lower is better. WER is useful, but it doesn’t tell the whole operational story, especially on live calls. If you want the broader failure pattern, this helps: why some AI voice agents get things wrong.
For Canadian businesses, the practical question isn’t “Can this model score well on benchmark audio?” It’s “What happens when a caller from Calgary is on Bluetooth in a work truck, or someone in Halifax calls from a concrete stairwell, or a customer in Brampton mixes English with Punjabi or Hindi names?”
Speech recognition accuracy drops sharply from clean studio audio to noisy phone calls
Why the phone line is the roughest surface your agent hears
A phone call strips away quality before the AI even gets a turn.
Phone audio is compressed. It’s often narrowband. Mobile calls can suffer handoffs between towers. Speakerphones add distance and room echo. Truck cabs add engine noise, road hiss, and reflective surfaces. Hard walls in a shop or lobby add reverberation. Cheap microphones smear consonants. A weak signal can clip the start or end of a word, which is exactly where many names and numbers become distinguishable.
So accents and background noise aren’t really one problem for an AI voice agent. They’re four problems stacked on top of each other: the channel, the environment, the speaker, and the content.
On a noisy call, the agent has to separate speech from background sound, reconstruct damaged audio, guess through compression artefacts, and still decide whether the caller said “fifteen” or “fifty.” On a website demo, none of that is happening.
There’s also timing. Live agents don’t get unlimited time to think. Realtime systems have to transcribe quickly enough to keep the conversation moving. That speed requirement can expose weaknesses you won’t see in offline benchmark clips.
For automated systems, contact-centre benchmarks generally need 90%+ accuracy. Around 85%+ may be workable for agent assistance, where a human is still supervising.
If your goal is full call handling, “pretty good most of the time” is not a safe standard.
Accents, dialects, and the Canadian reality
In Canada, speech recognition across accents isn’t an edge case. It’s the normal operating condition.
According to Statistics Canada language data, 19.4% of Canadians reported speaking more than one language at home in 2016, up from 17.5% in 2011. In practice, many of Canada’s biggest metro areas are linguistically dense and highly mixed. Think Toronto and Brampton. Surrey and Vancouver. Winnipeg. Halifax. Ottawa. Montreal among many others.
That matters because accent performance is not binary. It degrades gradually, and then it breaks suddenly when combined with noise, speed, and unfamiliar vocabulary. AssemblyAI’s 2026 ranges put heavily accented speech at 75% to 90% accuracy depending on model and conditions. That’s a big band.
Some callers speak slowly and clearly but with unfamiliar vowel patterns. Others speak fast, clip endings, or use local street names, business names, and surnames the model barely saw in training. Pronunciation clarity, pace, dialect, and voice characteristics all affect results.
So when a vendor tells you accents and background noise are “solved,” the honest answer is that they aren’t. Managed well, yes. Solved, no.
What you should look for is whether the system recovers gracefully. Does it confirm the important part? Does it use prior context? Does it ask a tight follow-up instead of restarting the whole exchange? Those behaviours matter more than a vendor saying “we handle accents.”
Code-switching is where weak systems show themselves
There’s one stress test that exposes model choice very quickly: code-switching, where the caller changes language mid-sentence.
That’s common in Canadian cities. A caller may book in English, switch to French for a street name, then give a Punjabi, Arabic, Mandarin, or Spanish surname. Or they’ll ask a question in English and confirm details in French. If you serve Canadian SMEs across six time zones, this isn’t rare traffic. It’s Tuesday.
And the model spread here is brutal. On average normalized WER across five language pairs in 2026, AssemblyAI Universal-3.5 Pro scored 7.69%, ElevenLabs Scribe v2 8.77%, Deepgram Nova-3 Multilingual 12.22%, and OpenAI GPT-4o Transcribe 44.58%. That’s roughly a six-fold spread between the best system and GPT-4o Transcribe on the same code-switching audio, based on the published benchmark tables.
This is why clean-English demo scores can mislead you. The model your vendor picked matters far more than the polished sample call on their homepage suggests.
If your callers switch between French and English, or regularly mix in other languages, you should be asking for a bilingual and multilingual voice agent that has been tested on mixed-language turns, not just separate monolingual flows.
The expensive errors are names, numbers, and places
WER gets attention because it’s easy to compare. But if you run a business phone line, entity accuracy is usually what hits your revenue.
Entity errors are mistakes on the details that actually drive the workflow: names, phone numbers, addresses, appointment dates, unit numbers, product codes. A transcript can look mostly fine and still fail the call if one digit is wrong.
On the Pipecat open realtime benchmark for actual agent conversations, the gap is clear.
A 3.55% versus 10.41% phone-number error rate is not a lab curiosity. It decides whether the callback happens. It decides whether the estimate gets sent to the right person. It decides whether your front desk has to spend the afternoon cleaning up bad records.
Same with names. If your business serves multilingual neighbourhoods in Surrey, Toronto, Edmonton, or Winnipeg, surname accuracy isn’t cosmetic. It affects trust on the call and data quality after the call.
A business phone ringing in a noisy front-counter environment during peak hours
Why a missed word is worse than a wrong word
This part gets missed in vendor decks.
Traditional WER treats substitutions, insertions, and deletions as equal. In live voice agents, they are not equal. A substitution is a wrong but plausible guess. A deletion is a missing word. And on a realtime phone call, a deletion can be worse.
Why? Because a deletion can create a hanging turn. The caller speaks, but the agent receives too little content to act on. Then the system pauses, the caller hears silence, and the conversation starts to feel broken. People don’t wait long in dead air. They repeat themselves, talk over the agent, or hang up.
That’s why accent and background-noise problems usually surface as awkward silence long before they surface as an obviously wrong transcript. The system didn’t confidently hear enough to move.
If you test a vendor, don’t just read the transcript after the call. Listen for turn-taking. Did the agent respond promptly? Did it ask for a repeat in a targeted way? Or did it freeze because a key word vanished?
What actually improves performance in the field
There are fixes that measurably help. Not slogans. Actual engineering choices.
1) Speaker isolation matched to the setting
Use voice focus or speaker isolation tuned to the environment. Near-field mode works for phones and headsets, where the speaker is close to the mic. Far-field mode fits rooms, counters, and drive-thrus. If a vendor uses the same audio front-end for every scenario, expect trouble.
This won’t save a caller standing in pure wind noise, but it can materially reduce background bleed from shops, cabs, and open offices.
2) Context carryover across turns
If a caller says an unusual surname once, the system should remember it and reuse that context later. Same for company names, street names, and product terms. Context carryover prevents the second mention from being mangled differently than the first.
3) Agent context passed into transcription
This one is powerful and underused. If the transcription model receives the agent’s own prior question, it gets a better frame for what the caller is likely saying next.
Across 20,000 voice-agent files, adding agent context cut WER by 10.2%, with fabrications down 18.3%, hallucinations down 17.2%, place entities down 15.5%, and short utterances down 13.7%. One team pairing agent context with prompting cut utterance error rate from 26% to 9% on production audio.
That’s not magic. It’s just giving the model the conversational setup it was missing.
4) Keyterm prompting with your vocabulary
Feed the system your real terms: product names, staff names, local streets, acronyms, building names, service categories, postal patterns. In a healthcare test, contextual prompting cut missed domain terms by 31%.
If your vendor hasn’t asked for this list, they’re leaving accuracy on the table.
5) Per-word confidence thresholds
Every important field should have a confidence check. A practical rule is to flag anything under roughly 0.85, then re-ask or route to a human. Don’t pretend a low-confidence phone number is “good enough.” Confirm it.
A good re-ask is short and specific: “I want to make sure I got your callback number right. Can you repeat the last four digits?” Not “Sorry, please repeat everything.”
6) Human handoff triggers
You need a clear threshold for escalation. Repeated low-confidence entities, multiple hanging turns, code-switching the system can’t stabilize on, or obvious signal breakup should trigger transfer or callback capture.
Good systems don’t bluff. They fail cleanly.
A five-call stress test you can run this week
If you’re buying for a Canadian business, run these calls before signing anything.
- Call from a moving vehicle on speakerphone. Listen for delayed responses, clipped first words, and whether the agent can still capture a phone number correctly.
- Call from your noisiest room at your busiest hour. Use the shop floor, reception desk, kitchen pass, or warehouse edge. Listen for whether background voices bleed into the transcript and whether the agent keeps the turn structure intact.
- Have someone with a strong accent book an appointment. Include a street name, date, and surname. Listen for whether the system confirms the risky fields instead of guessing through them.
- Switch language mid-sentence. Don’t announce it. Just do it naturally. Listen for whether the model tracks the change or collapses into repeats and silence.
- Read a 10-digit phone number and a hard-to-spell surname at normal speed. No exaggerated pacing. Listen for exact capture, smart readback, and whether one wrong digit slips through.
What you’re checking is not just “Did it answer?” You’re checking recovery behaviour. Did it ask precise follow-ups? Did it preserve context? Did it hand off when confidence dropped?
If you want a fuller version, use the 7-call protocol for testing a vendor before you sign.
And record your own results. Benchmark scores are a starting point, not a verdict. The only number that matters is the one measured on your audio, in your conditions.
The limits are real, and good vendors will say so
No vendor can fix a caller in a wind tunnel. Or a dropped signal that removes half the sentence. Or deliberate mumbling. Or a speakerphone left on the dashboard while the driver shouts over road noise.
Audio quality still matters. Microphone quality matters. Compression matters. Echo from hard surfaces matters. Speaking pace matters. So do numbers, dates, and proper nouns that sit outside the model’s usual vocabulary.
A trustworthy system doesn’t pretend otherwise. It detects uncertainty, asks for the missing piece, switches tactics, or hands the call to a person. That’s what competence looks like on the phone.
So the right question about an AI voice agent and accents and background noise was never “Can it handle everything?” It’s “How well does it handle the calls we actually get, and what does it do when the audio turns against it?”
Whether your callers are in downtown Vancouver, a farm outside Brandon, or somewhere in the four hours of time zones in between, that’s the standard worth holding a vendor to.
Curious how your own worst call would go? Hear it handle a call like yours and bring your hardest surname.