Definition · Updated 17 July 2026
What Is an AI Voice Agent? How Autonomous Phone AI Works
An AI voice agent is software that holds a spoken phone conversation on its own: it listens to the person, understands what they said, decides how to respond, and speaks back in real time, turn by turn, with no human on the line. It is a phone call you can have with a program, not a menu you press buttons through.
Mechanically it is a loop of three parts running continuously: speech-to-text (STT) transcribes what the caller says, a large language model (LLM) reasons and writes the reply, and text-to-speech (TTS) voices it — all streaming in parallel so the reply starts before the caller has finished their thought. Two more mechanisms make it feel like a conversation rather than a walkie-talkie: turn detection, which decides when the person has finished so the agent can answer, and interruption handling, which lets the caller cut in.
That conversational ability is the whole distinction. An IVR phone menu maps keypad tones to fixed options; an auto-dialer places calls and connects a human when someone answers; an AI voice agent has the conversation itself. The bar is high because human conversation is fast — the mean gap between speakers is roughly a fifth of a second — so a voice agent has to hear, think, and answer inside about one second to sound natural. The category is also large and growing: Grand View Research valued the broader conversational AI market at USD 11.58 billion in 2024, projected to reach USD 41.39 billion by 2030 at a 23.7% CAGR.
The real-time loop: STT → LLM → TTS
The dominant architecture for a production voice agent is a streaming pipeline of three models. Each stage passes partial output downstream the moment it has any, instead of waiting for the previous stage to finish. That overlap is what buys the speed:
- 1.Speech-to-text (STT / ASR). Transcribes the caller's audio into text as they speak, emitting interim results and then a finalized transcript. This is also where the agent figures out the caller has stopped talking.
- 2.Large language model (LLM). Reads the running transcript plus the agent's instructions and context, decides what to say or which tool to call, and streams the response token by token.
- 3.Text-to-speech (TTS). Turns the model's words back into audio, starting to speak the first phrase while later ones are still being generated.
The reason streaming matters is the latency budget. LiveKit's engineering guide to voice-agent architecture budgets roughly under 50 ms for audio transport, 100–200 ms for the first STT result, 200–400 ms for the LLM's first token, and 100–300 ms for the first audio out — landing end-to-end response under one second. A naive sequential design that waits for each stage to fully complete before starting the next instead produces two to four seconds of delay, which listeners immediately register as unnatural.
One second is not an arbitrary target; it is set by how people talk. The 2009 study “Universals and cultural variation in turn-taking in conversation” (Stivers et al., PNAS 106(26):10587–10592) measured the gap between conversational turns across ten languages and found a cross-language mean of just +208 ms, with the most common value near zero. Language-specific averages ran from +7 ms for Japanese to +469 ms for Danish, clustering within about a quarter-second either side of the mean. In the authors' words, the language means fall “within ≈250 ms either side of this cross-language mean, approximately the length of time it takes to produce a single English syllable.” A voice agent that answers within roughly that window sounds human; one that takes two seconds sounds like a machine.
There is a second architecture worth knowing. Instead of three separate models, a speech-to-speech model takes audio in and produces audio out directly. OpenAI's Realtime API, per its own documentation, offers “voice-to-voice interaction with the model, without an intermediate text-to-speech or speech-to-text step,” which “enables lower latency” and “gives the model more data to work with around the tone and inflection of voice input.” The trade-off is control: the chained STT-LLM-TTS pipeline lets you swap each model, read the transcript, and steer the wording; the single-model approach is faster and more expressive but more of a black box.
Turn-taking and interruption (barge-in)
The hardest part of a voice agent is not generating good words — the LLM does that. It is the timing: knowing the exact moment the person has finished so the agent can answer, and stopping the instant the person cuts back in. Get either wrong and the agent talks over people or leaves dead air, and the illusion collapses.
Turn detection usually combines voice activity detection (VAD), which classifies each moment of audio as speech or silence, with endpointing, which decides that a pause is long enough to count as the end of a turn. Deepgram's endpointing documentation describes the behavior precisely: the model “detects sufficiently long pauses that are likely to represent an endpoint in speech,” then finalizes the transcript and returns it with speech_final set to true. Tune the silence threshold too short and the agent interrupts; too long and it feels sluggish. It is the single most-tuned parameter in a voice stack.
Interruption handling — barge-in is the mirror image: when the caller starts speaking while the agent is still talking, the system must stop playback and cancel the reply in flight. OpenAI's Realtime API enables VAD-based turn detection by default and, in its own words, “will automatically truncate unplayed audio when there's a user interruption” — discarding the queued speech the caller never heard so the transcript and the audio stay in sync. Barge-in is ultimately a latency problem disguised as a UX problem: an agent that cannot detect the interruption and go quiet within a couple hundred milliseconds will trample the person it is supposed to be listening to.
Inbound vs outbound voice agents
The core loop is identical whichever direction the call goes; what changes is the trigger and the rulebook.
- Inbound voice agents answer calls that come to them — reception, support triage, order status, appointment lookups. They are the natural replacement for a legacy IVR: instead of “press 1 for sales,” the caller just says what they need.
- Outbound voice agents place the call — sales outreach, reminders, renewals, collections, follow-ups. The agent initiates contact at machine volume, which is exactly why the compliance surface below applies to outbound and not to inbound.
Outbound is the regulated direction. In Declaratory Ruling FCC 24-17 (CG Docket No. 23-362), adopted 2 February 2024 and released 8 February 2024, the FCC confirmed that calls using AI voice technology fall within the TCPA's existing prohibition on artificial or prerecorded voice messages, because “this technology artificially simulates a human voice.” So an outbound AI voice agent needs prior express consent where the TCPA (47 U.S.C. § 227) requires it, identification of the responsible party, a working opt-out, and it must respect quiet hours — telephone solicitations to residential subscribers only between 8 a.m. and 9 p.m. in the called party's local time. The statute's private right of action at § 227(b)(3) provides $500 per violation, which a court “may, in its discretion” increase to “not more than 3 times” that amount — so up to $1,500 — where the violation is willful or knowing; that uplift is a capped discretionary maximum, not an automatic multiplier. See the AI cold calling legal overview and TCPA quiet hours for the detail.
Voice agent vs IVR vs dialer
“AI voice agent” gets used loosely for anything that makes automated phone calls, but three categories are genuinely different technologies doing different jobs:
| System | What it does | How it “talks” |
|---|---|---|
| IVR (interactive voice response) | Routes an inbound caller through a fixed decision tree | Plays pre-recorded prompts; caller responds by keypad tone or a narrow speech grammar — no open conversation |
| Auto / predictive dialer | Places outbound calls through a list at volume and connects a live human agent when someone answers | Does not converse at all — it moves calls to people; a human does the talking |
| AI voice agent | Holds the whole conversation — inbound or outbound — and can take actions mid-call | Open-ended, real-time dialogue via the streaming STT-LLM-TTS loop, with turn-taking and barge-in |
The dialer distinction also carries a legal edge worth stating precisely. In Facebook, Inc. v. Duguid (592 U.S. 395, decided 1 April 2021), the U.S. Supreme Court held 9–0 that an “automatic telephone dialing system” under the TCPA must have the capacity to store or produce telephone numbers “using a random or sequential number generator.” A voice agent that dials specific, known numbers from a CRM list generally does not meet that definition, so it is typically not an autodialer in the ATDS sense — but that is a separate question from the artificial-voice rules in FCC 24-17, which apply to an AI voice agent regardless of how the number was selected. Do not conflate the two: escaping the ATDS definition does not exempt an AI voice from the TCPA's artificial-or-prerecorded-voice restrictions. For the full comparison, see AI voice agent vs AI dialer.
Where Veera fits
Veera is a working outbound AI voice agent — an AI Business Aide built around the call. Live today: it places the call itself in 42 languages with native voice synthesis via Cartesia Sonic — including nine native Indian languages (Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, and Punjabi) alongside 33 global languages, with multilingual voice AI that switches language when the conversation does. A human can type a mid-call instruction or trigger a supervisor takeover without hanging up, and the agent writes Smart Notes — summary, decisions, and action items — straight onto the contact record inside Veera.
Around the call, three more capabilities are live: in-call WhatsApp document delivery (a brochure or quote sent during the conversation itself), verified-lead discovery, and AI lead scoring that ranks each contact hot / warm / cold with a written reason.
Stated plainly, so you can plan around it: Veera syncs into GoHighLevel and HubSpot and does not replace your CRM. Two-way contact sync and pipeline-stage mapping are live today, and Veera reads deals and their stages from the CRM — the CRM stays the system of record. Writing call outcomes back into GoHighLevel and HubSpot, pushing deals, and activity sync are built and activating rather than usable today. The same is true of the SMS, WhatsApp, and email send channels (beyond the in-call WhatsApp delivery above), multi-step sequences, the unified inbox, and analytics dashboards — built and activating, not usable yet. Calendar and meeting booking is on the roadmap, not built. And Veera is an outbound agent: it places calls, it does not yet answer inbound ones.
On compliance, Veera enforces controls in the send path rather than reporting on them after the fact: TCPA quiet hours are checked per call and are timezone-aware to the called party's location, opt-outs are suppressed before the next call, and erasure is carried out when a contact asks to be forgotten. That is a description of enforced controls, not a certification — compliance also depends on your consent practices and jurisdictions. Veera is free to start; the calling side is covered in the AI cold calling software hub and the languages in multilingual AI calling.
Frequently asked questions
What is an AI voice agent?
An AI voice agent is software that conducts a spoken phone conversation on its own — it listens to the person, understands what they said, decides how to respond, and speaks back in real time, turn by turn, with no human on the line. Under the hood it runs a continuous loop of three components: speech-to-text (STT) transcribes the caller, a large language model (LLM) reasons and generates the reply, and text-to-speech (TTS) voices it, all streaming in parallel so the exchange feels like a call rather than a voicemail. That conversational ability is what separates a voice agent from an IVR phone menu, which only maps keypad presses to fixed options, and from an auto-dialer, which places calls and hands a connected human to a live rep but does not itself talk.
How does an AI voice agent work in real time?
It chains speech-to-text, a language model, and text-to-speech into a streaming pipeline where each stage passes partial output downstream instead of waiting for the previous stage to finish. LiveKit's engineering guide budgets roughly under 50 ms for audio transport, 100 to 200 ms for the first speech-to-text result, 200 to 400 ms for the language model's first token, and 100 to 300 ms for the first audio out — about one second end to end. A sequential design that waits for each stage to complete instead produces two to four seconds of delay, which listeners register as unnatural. The target exists because human conversation is fast: the 2009 PNAS study "Universals and cultural variation in turn-taking in conversation" measured a cross-language mean gap between turns of just +208 ms across ten languages, so a voice agent has to answer within roughly that window to feel human.
How does an AI voice agent know when to talk and handle interruptions?
Two mechanisms. Turn detection decides when the person has finished speaking so the agent can respond, usually via voice activity detection (VAD) plus endpointing — Deepgram's streaming speech-to-text, for example, finalizes a transcript after it detects a configurable stretch of silence and marks it speech_final. Interruption handling, or barge-in, does the opposite: the instant the person starts talking over the agent, the system stops playback and cancels the pending response. OpenAI's Realtime API enables VAD-based turn detection by default and, in its own words, "will automatically truncate unplayed audio when there's a user interruption." Barge-in is really a latency problem: an agent that cannot stop within a couple hundred milliseconds talks over people and feels robotic.
What is the difference between an AI voice agent, an IVR, and a dialer?
An IVR (interactive voice response) is a menu: it plays fixed prompts and routes the caller by keypad tone or a narrow speech grammar, with no open-ended conversation. A dialer places calls — a predictive or auto-dialer works through a list at volume and connects a live human agent when someone answers; it moves calls, it does not hold them. An AI voice agent holds the conversation itself. The dialer distinction also has a legal edge: in Facebook, Inc. v. Duguid (2021) the U.S. Supreme Court held 9-0 that an "automatic telephone dialing system" under the TCPA must have the capacity to store or produce numbers using a random or sequential number generator, so a voice agent dialing specific numbers from a CRM list generally is not an autodialer under that definition — though the TCPA's separate artificial-or-prerecorded-voice rules still apply to it.
What is the difference between inbound and outbound voice agents?
Inbound voice agents answer calls that come to them — reception, support, order status, appointment lookups — and are the natural replacement for a legacy IVR. Outbound voice agents place the call: sales outreach, reminders, renewals, follow-ups. The core STT-LLM-TTS loop is the same, but the trigger and the rules differ. Outbound calling carries a compliance surface inbound does not: in the US the FCC's Declaratory Ruling FCC 24-17, adopted 2 February 2024, confirmed that AI-generated voices are "artificial" under the TCPA, so outbound AI calls need the consent, identification, opt-out, and 8 a.m. to 9 p.m. quiet-hours rules that govern any artificial-voice call. Veera is an outbound voice agent.
What can Veera's AI voice agent do today, and is it free?
Veera is free to start and is a working outbound AI voice agent. Live today: it places the call itself in 42 languages with native voice synthesis via Cartesia Sonic — including nine native Indian languages (Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, and Punjabi) — lets a human type a mid-call instruction or take over the conversation mid-call, writes Smart Notes (summary, decisions, and action items) onto the contact record inside Veera, delivers a document over WhatsApp during the call, and runs verified-lead discovery and AI lead scoring. It syncs into GoHighLevel and HubSpot with two-way contact sync and pipeline-stage mapping and reads deals and their stages; it does not replace your CRM. Writing call outcomes back into those CRMs, the SMS, WhatsApp, and email send channels, multi-step sequences, the unified inbox, and analytics dashboards are built and activating rather than usable today, and calendar booking is on the roadmap.
This definition is published by Veera, which makes an outbound AI voice agent. Technical and legal facts on this page are cited to their primary sources — the LiveKit voice-agent-architecture guide, the OpenAI Realtime API docs, Deepgram endpointing docs, the 2009 PNAS turn-taking study, Facebook v. Duguid, FCC Declaratory Ruling FCC 24-17, and 47 U.S.C. § 227 — and are current as of 17 July 2026. Nothing here is legal advice; consult counsel for your jurisdiction and consent practices.