What does the LLM do in an AI voice agent?
The large language model, or LLM, is the decision-making part of an AI voice agent. It reads the live transcript, the business's instructions and any facts it has been given, then decides what to say next and whether to take an action such as booking a slot. Speech recognition and text-to-speech handle hearing and speaking; the LLM handles understanding and choosing the reply.
The part between hearing and speaking
Speech recognition turns audio into text and text-to-speech turns text into audio. Everything in between is the language model's job. On every turn it receives the conversation so far and produces the next reply, much like a chat assistant, but with replies kept short and conversational because they will be spoken aloud.
What the LLM receives on each turn
- —A system prompt: the business's instructions on goal, tone, language and limits.
- —Reference facts: prices, timings, locations, policies or product details.
- —The conversation history: every turn so far, as text.
- —Tool definitions: actions it may trigger, such as checking a calendar or writing to a sheet.
- —Caller context: details known before the call, such as the lead's name or the form they filled.
What it decides
From that input the model chooses one of a few moves: answer a question, ask the next qualifying question, confirm a detail, call a tool or end the call politely. It also has to judge intent, for example whether 'abhi busy hoon' means 'call me later' or 'not interested'.
Trade-offs in choosing a model
Bigger models reason better and follow complex instructions more reliably, but respond more slowly. On a phone call every hundred milliseconds counts, so voice platforms balance intelligence against speed. Some use a fast model for most turns; others narrow the task carefully so a faster model is enough.
Why the LLM is not the whole story
It is tempting to judge a voice agent by the model behind it, but callers experience the whole system. A strong model behind slow turn detection, a robotic voice or a poor phone connection still makes a bad call. The quality of the instructions matters as much as the model: vague prompts produce vague calls.
Outpero does not publish the names of the models in its stack. What it publishes is the outcome: 0.8 to 1.3 seconds on average end-to-end latency, seven live languages and the ability to book calendar slots, update spreadsheets, call custom APIs and send WhatsApp messages during a call.
Want to hear an AI voice agent handle this on a real call?
Follow-up questions
Does the LLM learn from my calls automatically?
+
In most voice agent platforms, not in the sense of retraining itself after each call. The model's knowledge is fixed; what changes its behaviour is the instructions and facts you give it. When transcripts reveal a recurring problem, you update the instructions and the change applies to all future calls, which is faster and more predictable than retraining.