How AI Voice Agents Work: Speech, LLM, Voice and 0.8–1.3s Latency

The short version

When a customer says 'kal subah ka appointment mil sakta hai?' to an AI voice agent, five things happen in about a second and a half. The phone network delivers the audio. The agent decides the customer has finished speaking. Speech recognition converts the audio to text. A language model reads the text, checks your instructions and calendar, and writes a reply. A voice model reads that reply aloud, and the audio travels back down the line.

Each of those steps is well understood on its own. The craft is in making them overlap, in handling interruptions, and in doing all of it with Indian languages, Indian phone lines and callers who change language in the middle of a sentence.

Step 1: Speech recognition (ASR)

Automatic speech recognition, or ASR, converts the caller's audio into text. Phone audio is harder than studio audio. It is narrow-band, it passes through a mobile network that drops fragments, and the caller may be on a road, in a market or in a clinic corridor.

In India the recogniser also has to cope with accents that vary every few hundred kilometres, with numbers spoken in a regional language, and with code-switching. A Hyderabad caller might say 'mee clinic lo evening appointment undha' and then add 'around six o'clock' in the next breath. A recogniser built for English alone often turns that into noise, which is why regional-language voice AI needs speech models that are evaluated on mixed-language speech and not just on each language separately.

ASR works in a stream. It does not wait for the sentence to end; it produces partial text as the caller speaks, then finalises the words once the caller pauses. That early text is what lets the next step start sooner.

Step 2: The language model decides

A large language model, or LLM, is the brain. It receives the conversation so far, your instructions (the script and rules), the knowledge you supplied (prices, timings, eligibility) and the caller's latest sentence. It then decides on one of two things: say something, or do something.

Saying something is generating the next reply, and on a phone call replies must be short. One or two sentences, in the language the caller is using. Doing something is called an action or function call, and this is where a voice agent becomes useful instead of just chatty.

• —Book a real slot in a calendar and read it back to the caller.
• —Read or write a row in a spreadsheet, live, during the call.
• —Call your own API, for example to check an order or a policy status.
• —Send a WhatsApp message with a location pin or a brochure during or just after the call.
• —Transfer the call to a person on your team when the situation needs one.

Each action is opt-in per AI employee, so an appointment-booking agent can book but a lead-qualifying agent cannot unless you allow it. Good agents also know their limits: when they do not have the answer they should say so and offer a callback or a handover rather than guess.

Step 3: Text-to-speech (TTS) gives the reply a voice

Text-to-speech converts the model's reply into audio. Older systems sounded like a robot reading a notice, with flat rhythm and a pause between words. Modern neural voices carry intonation, breath and emphasis, and in Indian languages they must also pronounce the English words a speaker would naturally mix in: 'EMI', 'appointment', 'site visit', 'OTP'.

Outpero's voice engine is called Swara, and it offers two tiers: Standard at ₹5 a minute, with expressive voices for everyday sales and support, and Premium at ₹6.8 a minute, with the most lifelike voices for calls where the voice is the brand. The tier changes the voice, not the languages: the price is the same in all seven.

TTS also streams. It begins speaking the first sentence of the reply while the model is still writing the second. Without that, the caller would wait for the whole paragraph to be generated before hearing anything.

Streaming: why the three steps overlap

If the three steps ran one after another, each waiting for the previous to finish completely, a typical turn would take three to five seconds of silence. That is long enough for a caller to say 'hello? hello?' and hang up. Streaming overlaps the stages like a relay race where each runner starts as soon as the baton moves.

• —ASR hands partial text to the model while the caller is still finishing.
• —The model streams its reply word by word instead of delivering a finished paragraph.
• —TTS starts speaking as soon as it has a first clause, then keeps generating behind it.

The result is that the caller hears the start of the answer well before the whole answer exists. This is the biggest single reason a modern agent feels responsive.

Voice activity detection and endpointing

Voice activity detection, or VAD, is a small model that answers one question many times a second: is someone speaking right now? Endpointing builds on it and answers a harder question: has the speaker finished their turn?

This matters more than it seems. Humans leave an average gap of only about 200 milliseconds between turns, according to a widely cited 2009 cross-language study of conversation by Stivers and colleagues. A voice agent that waits two full seconds of silence to be safe feels slow. One that jumps in after 300 milliseconds interrupts every caller who pauses to remember a name.

Indian callers add complications. People pause mid-sentence to think of a number, say 'haan... woh...' while looking for a word, or read out a ten-digit mobile number in groups with pauses between each group. A good endpointer looks at more than silence: it considers whether the sentence is grammatically complete, whether the caller's tone fell, and how the pace of this person's speech has been so far.

Barge-in: when the caller interrupts

Real callers interrupt. They say 'haan haan, samajh gaya' when the agent is mid-explanation, or they correct something halfway. Barge-in is the ability to detect that the caller has started speaking while the agent is talking, stop the agent's audio at once, and listen.

Done badly, barge-in has two failure modes. The agent can talk over the caller and ignore them, which is the IVR experience everyone dislikes. Or it can stop every time the caller coughs, a vehicle honks or a background television plays. Handling it well means distinguishing speech from noise and recognising the short acknowledgements ('hmm', 'ok', 'haan') that mean the caller is following, not taking over.

The latency budget: where the 0.8 to 1.3 seconds goes

Latency here means the time from the caller finishing a sentence to the first sound of the agent's reply. Outpero's end-to-end conversational latency is 0.8 to 1.3 seconds on average. The table below shows an illustrative split of where time goes in a pipeline like this one; it is a teaching breakdown, not a measurement of any specific call.

Two details are worth noticing. First, the biggest items are the model and the endpointing wait, which is why most engineering effort goes into those two. Second, the telephone network contributes time that no software can remove; a call routed through three carriers is slower than one that stays on a local, well-connected line.

Below roughly a second, many callers begin to feel the agent is too eager. Above two seconds, they begin to say 'hello?'. 0.8 to 1.3 seconds on average sits in the range that most callers read as a person thinking before they answer.

After the call: what is recorded and returned

Everything the agent heard and said becomes data. For every call, Outpero stores the recording and full transcript, writes a short summary, scores the sentiment and fills the custom fields you asked for, such as budget, preferred time or reason for cancelling. All of it appears in the dashboard (call logs, leads database, analytics, CSV export) and can be posted to your webhook, so your CRM or sheet updates without anyone retyping.

The call log also shows the credits each call used, so you can see the cost of every conversation. Unanswered calls are not billed. Credits are prepaid at 1 credit per rupee and never expire, and 50 free credits come with signup.

Inbound, outbound and caller memory

The pipeline is identical in both directions; what differs is who speaks first and why.

Inbound agents have one extra capability: caller memory. When a returning customer rings, the agent recognises them and picks up from the previous conversation instead of starting from zero. That small thing changes how a caller feels about calling a business at all.

A channel is one call at a time on your number. A number costs ₹649 a month and includes one channel; each extra channel is ₹649 a month, so five channels means five calls at once.

Voice broadcast is not an AI conversation

Voice broadcast is often confused with an AI voice agent because both use a phone and a synthetic voice. They are different tools.

Use a broadcast when everybody needs to hear exactly the same words, such as 'your fee instalment is due on Friday'. Use an AI agent when the customer is likely to ask a question or the answer decides what happens next.

What can still go wrong

• —Misheard numbers. Phone numbers, amounts and dates are the most common ASR errors. A good script repeats them back and asks for confirmation.
• —Invented answers. A language model will sometimes produce a confident wrong reply when it lacks the facts. The remedy is a knowledge base, narrow instructions and a clear rule to offer a callback when unsure.
• —Noisy lines. Heavy background noise degrades recognition and triggers false barge-in. Short, confirming questions help.
• —Callers who want a person. Offer transfer to your team and make it easy; forcing someone to argue with an AI is bad for them and for you.

Frequently asked questions