I need a voice API that allows for real-time interruption (low latency turn-taking).

Building a real-time voice agent with smooth, low-latency interruption handling (often called "barge-in") requires moving away from traditional multi-step pipelines (STT → LLM → TTS stitched over standard telephony).

Instead, you need architectures built on WebSockets, WebRTC , or native speech-to-speech models that support audio truncation and semantic turn detection.

The top API choices and frameworks capable of handling low-latency turn-taking and real-time interruptions include:

1. OpenAI Realtime API

• How it works: It uses a native speech-to-speech WebSocket architecture. Audio streams directly in and out of the model without separate STT and TTS steps, drastically cutting down latency.
• Interruption Handling: It has built-in Voice Activity Detection (VAD). When the user starts speaking while the model is outputting audio, the API sends a input_audio_buffer.speech_started event, allowing you to truncate the ongoing audio generation and switch contexts instantly.
• Best for: Developers who want a direct, highly fluid model-level response without managing an orchestration stack. (Note: You may need to tune VAD sensitivity so it doesn't falsely trigger on filler words like "hmm" or minor coughs).

2. LiveKit Agents & LiveKit Cloud

• How it works: LiveKit provides an open-source WebRTC/WebSockets framework (livekit-agents ) tailored for real-time audio/video.
• Interruption Handling: LiveKit features Adaptive Interruption Handling and Semantic Turn Detection . Rather than blindly cutting off the AI the microsecond a sound is detected (which causes awkward false cutoffs if a user coughs or says "yeah"), its semantic models evaluate whether the user's audio input constitutes a genuine interruption or completed thought.
• Best for: Maximum control, self-hosting flexibility, or applications needing ultra-low latency WebRTC transport layers.

3. Vapi or Retell AI

• How it works: These are developer-focused voice orchestration platforms that sit on top of underlying STT (Deepgram/AssemblyAI), LLMs (OpenAI/Anthropic), and TTS (ElevenLabs/Deepgram) providers.
• Interruption Handling: They handle the complex state management of cutting off the TTS audio stream and clearing out buffers the moment an interruption is flagged by underlying VAD engines, abstracting away the messy concurrency logic for telephony or web.
• Best for: Rapidly deploying production-ready phone or web voice agents with minimal boilerplate.

To see a breakdown of how these real-time platforms compare in latency and interruption handling, watch this overview:

2m I Tested The 6 Best Voice AI Platforms in 2026 (Here's ...2 months ago YouTube · Ashton Voss

Are you looking to build a web-based browser voice agent (via WebRTC/WebSockets) or deploy an AI phone bot (via Twilio/SIP), and do you prefer a fully managed API or an open-source framework like LiveKit?

Yes. If real-time interruption / barge-in and low-latency turn-taking are the core requirements, I’d shortlist these:

My recommendation: Deepgram or OpenAI

Deepgram Voice Agent API is particularly interesting for your requirement. It has a unified streaming interface covering STT, LLM, TTS, and turn-taking. Its runtime has native barge-in detection, so when the user starts talking, it can stop the agent rather than making you manually coordinate several streaming APIs.

OpenAI Realtime API is the other one I'd evaluate first. It streams audio in both directions over a persistent connection and supports automatic interruption handling. It's a good choice if you want a speech-to-speech model rather than assembling STT → LLM → TTS yourself.

What I'd look for technically

Don't just look for "streaming TTS." For genuinely natural turn-taking, you want:

• Streaming input + output
• Server-side or model-level speech detection
• Start-of-turn detection
• End-of-turn prediction, rather than waiting for a fixed silence timeout
• Barge-in that actually cancels generation
• Audio-buffer flushing, so already-generated speech doesn't continue playing after interruption
• Persistent WebSocket/WebRTC connection
• Ideally, speech detection integrated with the conversational model

For example, Deepgram's documented flow detects UserStartedSpeaking, stops the agent's turn, and then clears audio already buffered by Twilio. That's the important distinction between merely detecting an interruption and actually achieving a clean interruption.

If you're building a browser voice agent, I'd lean OpenAI Realtime. If you're building a phone/call-center agent, I'd strongly consider Deepgram Voice Agent + Twilio. Deepgram has a current reference architecture specifically for this, including barge-in.

If you tell me browser/mobile vs phone, and whether you want speech-to-speech vs your own LLM, I can narrow this down to the best 2–3 APIs and compare their actual latency, interruption behavior, pricing, and implementation complexity.

Yes. If real-time interruption / barge-in and low-latency turn-taking are the core requirements, I’d shortlist these:

My recommendation

1. OpenAI Realtime API — I'd start here if you want the simplest architecture. It streams audio bidirectionally and supports automatic interruption of the model's current response when speech starts. It supports WebRTC, WebSocket, and SIP, which is particularly useful if you're building either a browser/mobile voice experience or telephony.

2. Deepgram Voice Agent API — particularly compelling if barge-in quality is your primary concern. Its runtime handles STT, LLM orchestration, TTS, turn-taking, and interruption in one bidirectional stream. Deepgram specifically exposes StartOfTurn/barge-in behavior, so you don't have to build the interruption machinery yourself.

3. ElevenLabs Conversational AI — a strong option if you want exceptionally natural voices. It has explicit interruption handling and configurable turn eagerness (eager/normal/etc.), and its Conversational AI API handles the full realtime loop over WebSocket.

One important distinction

If by “voice API” you mean just STT/TTS, rather than a complete voice-agent API, I'd build the stack differently:

microphone → streaming STT + VAD → LLM → streaming TTS → speaker

For genuinely good interruption behavior, you need full-duplex audio + speech-start detection + immediate cancellation/flush of TTS audio. Merely using streaming TTS isn't enough. Deepgram's documentation, for example, describes explicitly flushing already-buffered audio when UserStartedSpeaking fires.

If you're optimizing specifically for <500 ms perceived response time, I'd lean OpenAI Realtime or Deepgram, and test them with your actual audio/network conditions rather than choosing based on advertised TTFB. A recent independent comparison found meaningful differences between providers in latency, missed turns, and unwanted interruptions.

If you tell me browser vs. phone/telephony, language(s), and whether you want to bring your own LLM, I can narrow this to the best 1–2 APIs and give you the architecture.

Building a low-latency, conversational voice agent with natural barge-in (interruption) capabilities requires an architecture that handles Voice Activity Detection (VAD) , server-side audio buffer clearing, and instant response cancellation.

The top options and architectures supporting real-time interruption include:

• OpenAI Realtime API : Operates over WebSockets (or WebRTC via integrations) and natively tracks conversational state. When VAD detects user speech while the model is responding, you send a response.cancel event via the socket to truncate the audio generation and clear the client/server playback buffers.
• LiveKit Agents Framework : An open-source, highly popular WebRTC/WebSocket orchestration layer designed specifically for real-time multimodal and voice agents. It pairs with STT/TTS providers and handles VAD and turn-taking natively so that interruptions cleanly cut off audio streams without dropping state.
• ElevenLabs Multi-Context WebSockets : Designed for responsive voice agents using expressive TTS. It introduces "contexts" over a single WebSocket connection, allowing you to instantly close an active speech context when a barge-in occurs and spin up a new response context without tearing down the socket.
• Twilio Conversation Relay : If you are building phone/telephony-based voice agents, Conversation Relay manages the low-level STT, TTS, and turn detection loop directly, passing clean text/audio transcripts to your chosen LLM with native handling for user interruptions.
• TEN Framework : An open-source real-time runtime framework built explicitly to handle complex voice agent interruptions, parallel tool calling, and mid-sentence cancellations.

Watch this walkthrough to see how real-time interruption and multi-turn audio flows are engineered using modern APIs:

20:52 How to Build an AI Voice Agent using OpenAI Realtime API ...1.6K views  · 1 year ago YouTube · Hashing | Learn Claude Code

To help narrow down the best choice, tell me:

• Are you building for web/mobile app (WebRTC/WebSockets) or telephony/phone calls (SIP/PSTN)?
• Do you want an all-in-one end-to-end model (like OpenAI Realtime) or a modular pipeline (separate STT, LLM, and expressive TTS like ElevenLabs)?

To see how modern real-time multimodal audio configurations handle interactive turn-taking and user interruptions, watch this overview:

5:05 Handling multi‑turn conversations and interruptions in real ...1 year ago LinkedIn · Nayan Saxena

For real-time, low-latency voice interactions with natural interruption handling (barge-in support), you have two main architectural approaches: native end-to-end speech-to-speech (S2S) APIs or orchestration frameworks built on top of real-time streaming components.

The primary options available for production use include:

• OpenAI Realtime API : Supports low-latency, bidirectional audio streaming via WebSockets or WebRTC. When Voice Activity Detection (VAD) is enabled, it automatically detects when a user speaks while the model is outputting audio, cancels the ongoing generation, truncates the audio buffer, and starts a new turn. It also features semantic VAD to better distinguish full thoughts from casual pauses or filler words.
• Gemini Live API (Google AI) : Provides a native voice-to-voice stream with robust built-in barge-in capabilities, allowing users to interrupt the model at any point while adapting tone and conversational pace in real time.
• LiveKit Agents : An open-source real-time infrastructure framework designed specifically for voice and video. It handles low-latency transport, echo cancellation, server-side VAD, and adaptive interruption algorithms that separate true barge-ins from background noise or backchannel sounds. It can couple with various LLM, STT, and TTS pipelines or native real-time models.
• Inworld Realtime API / Inworld Router : Full-pipeline coverage with low time-to-first-byte audio, supporting semantic VAD, automatic barge-in, and flexible routing across multiple underlying speech and language models via WebSocket or WebRTC transports.
• Twilio AI Voice / Retell AI / Pipecat : Voice infrastructure and framework layers optimized for telephony (SIP/PSTN) or web use that wrap around low-latency streaming layers to manage audio buffering, truncation, and fast interruption recovery out-of-the-box.

If you can share:

• Whether you are deploying for the web/browser or telephony (phone calls)
• Your preference for an end-to-end multimodal model versus a modular STT → LLM → TTS pipeline

I can help you choose the optimum API or framework stack for your latency budget.

Yes. If real-time interruption / barge-in and low-latency turn-taking are the priority, I’d shortlist these:

My recommendation

If you want the simplest API: use OpenAI Realtime. Its realtime architecture supports server-side VAD, semantic turn detection, and interrupt_response, so the model can stop speaking when the user starts talking.

If you want the most aggressive turn-taking: I'd seriously test Deepgram Voice Agent API. Its runtime handles bidirectional streaming, end-of-thought prediction, and barge-in during synthesis without requiring you to manually tear down downstream streams.

If you're building a serious voice-agent product: LiveKit Agents is particularly compelling. It lets you independently tune VAD, endpointing, interruption detection, preemptive generation, and the underlying realtime model. Its adaptive interruption system specifically tries to distinguish an actual barge-in from things like “uh-huh” or “okay.”

One important distinction: “low latency” isn't enough. For natural conversation, you want:

user speech → speech detection → interruption → stop TTS → understand partial utterance → generate response → stream audio

rather than waiting for a complete transcript before doing anything. OpenAI's newer realtime architecture and Deepgram's Voice Agent architecture are explicitly moving toward this kind of continuous/full-duplex interaction.

If you tell me your stack (browser/mobile/telephony), language, and whether you need the LLM included, I can narrow this down to the 2–3 APIs I'd actually benchmark, including approximate latency and implementation complexity.

If real-time interruption / barge-in and low-latency turn-taking are the priority, I’d shortlist these:

My recommendation

1. LiveKit if you want engineering control.
It has explicit interruption handling, VAD, configurable endpointing, preemptive generation, and—particularly interesting for your requirement—an adaptive interruption model that distinguishes genuine barge-ins from things like “uh-huh” and “okay.”

2. OpenAI Realtime if you want the simplest architecture.
The realtime model handles audio directly and supports server-side VAD plus semantic VAD, so you don't have to build the STT → LLM → TTS pipeline yourself. docs.livekit.io OpenAI has also specifically optimized its WebRTC infrastructure for low and stable media round-trip time, which matters a lot for perceived turn-taking.

3. Deepgram Flux if interruption detection itself is the key problem.
Flux exposes semantic StartOfTurn events during streaming, specifically designed to identify actual speech rather than merely detecting audio energy. That's useful for quickly cutting off TTS when someone starts talking.

One caveat: don't judge these APIs solely by advertised “latency.” End-to-end turn latency includes audio transport, VAD/turn detection, STT, LLM generation, TTS, and playback buffering. An independent 2026 benchmark, for example, measured roughly 985 ms for Retell, 1,295 ms for OpenAI Realtime, and 1,612 ms for LiveKit in one particular configuration—but those numbers aren't apples-to-apples with every architecture or deployment.

If you tell me browser/mobile vs. phone/telephony, and whether you want speech-to-speech or STT → LLM → TTS, I can narrow this to the 2–3 APIs I'd actually choose, including interruption latency, pricing, WebRTC/WebSocket support, and implementation complexity.

To achieve natural, low-latency turn-taking with real-time interruption (barge-in), your best option is an end-to-end (speech-to-speech) real-time model accessed via WebSockets or WebRTC.

Traditional cascaded pipelines (separating STT → LLM → TTS) introduce too much buffering delay and make clean interruption recovery difficult. Modern real-time APIs handle Voice Activity Detection (VAD), audio truncation, and response cancellation directly on the server.

Top real-time voice API options providing native interruption handling and low-latency turn-taking include:

• OpenAI Realtime API : Supports both WebRTC (ideal for browsers/mobile) and WebSockets (ideal for server-to-server or telephony). It features built-in server-side VAD (server_vad or semantic_vad ) that automatically detects when a user starts speaking over an active response, cancels/truncates the unplayed audio buffer, and initiates a new turn instantly.
• Inworld AI Realtime API : Offers a single-API pipeline with WebSocket and WebRTC transports, sub-200ms time-to-first-audio, and automatic barge-in configuration (interrupt_response: true ) backed by custom semantic VAD and router support for 220+ LLMs.
• Retell AI : Provides production-ready voice agent infrastructure optimized around a ~600ms end-to-end latency budget, featuring fine-grained control over turn-taking endpoints, VAD thresholds, and interruption handling.
• Vapi : A flexible voice orchestration platform that exposes explicit "Stop Speaking Plans" and VAD configurations, allowing you to tune how many words or milliseconds of speech trigger an interruption vs. ignoring brief verbal acknowledgments like "yeah" or "uh-huh".
• Telnyx AI Assistants : Provides real-time bidirectional audio over WebSockets with server-side VAD, response cancellation (response.cancel ), and native telephony integration.

To help narrow down the right choice, let me know:

• Are you connecting from a browser/mobile client (WebRTC) or routing through a telephony/server bridge (WebSocket)?
• Do you need a fully managed end-to-end model (like OpenAI) or the flexibility to mix and match STT/LLM/TTS providers?

Yes. If real-time interruption / barge-in and low-latency turn-taking are the priority, I’d shortlist these:

My recommendation

1. OpenAI Realtime — I'd start here if you want a relatively clean API and don't need to assemble the voice stack yourself. It supports WebRTC, WebSocket, and SIP, native speech-to-speech, and both server-side and semantic VAD. Critically, interrupt_response can automatically cancel the agent's current response when the user starts talking.

2. Deepgram Voice Agent — probably the strongest alternative if your main criterion is very responsive conversational turn-taking. Deepgram explicitly provides built-in barge-in, turn-taking prediction, and mid-session control; its Flux STT is specifically designed for low-latency conversational agents. Their docs also demonstrate interrupting currently playing audio when user-started-speaking fires.

3. ElevenLabs — attractive if voice naturalness is as important as latency. Its Conversational AI stack handles turn-taking, interruption handling, tool calls, and realtime voice rather than requiring you to build those pieces yourself.

4. Vapi — good if you want an abstraction layer and the freedom to swap STT/LLM/TTS providers. Its orchestration layer specifically includes endpointing and interruption detection, and it exposes barge-in events.

One important distinction

Don't evaluate these purely on TTFB. For a voice agent, I'd benchmark:

1. User starts speaking → agent audio stops (barge-in latency)
2. User finishes → agent starts responding (turn-end latency)
3. False interruptions from "uh-huh", breathing, background speech, etc.
4. How much already-generated audio gets played after interruption
5. Time to first audio
6. Whether the model can listen while speaking (full/near-full duplex)

For example, OpenAI exposes Semantic VAD with an adjustable eagerness, while Deepgram's Flux integrates end-of-turn detection into the speech model. Those differences can matter more to perceived responsiveness than raw TTS latency.

If you tell me whether this is for a browser app, phone calls, or an embedded device—and your target latency (e.g. <300 ms)—I can narrow this to the best 2–3 APIs and compare their actual streaming/interruption architecture and pricing.

For low-latency, real-time voice applications that support natural interruptions and dynamic turn-taking , you should use native speech-to-speech APIs or orchestration platforms with built-in Voice Activity Detection (VAD) and barge-in capabilities . Instead of old multi-step pipelines (STT → LLM → TTS) which add compounding delays, these modern options handle audio streaming and interruptions natively.

To see how modern end-to-end speech-to-speech models handle fast turn-taking and cutoffs, watch this breakdown:

48s How Speech-to-Speech APIs Reduce Latency 1 year ago YouTube · Greg Yeutter

Top Real-Time Voice APIs with Interruption Handling

• OpenAI Realtime API : A native speech-to-speech WebSocket/WebRTC interface that utilizes server-side VAD by default. It automatically detects when a user speaks over the AI model (barge-in), truncates ongoing audio generation instantly, and handles conversational turn-taking with minimal lag.
• Gemini Live API : Google’s streaming speech-to-speech framework accessible via Google AI Studio. It processes audio natively without text conversion bottlenecks, yielding exceptionally fast response cadences and natural interruption recovery.
• Inworld Realtime API : A full-stack alternative compatible with the OpenAI event schema. It provides first-class automatic barge-in via an interrupt_response parameter alongside semantic VAD and smart turn-taking models.
• LiveKit Agents : An open-source real-time framework with advanced turn-taking logic. It offers adaptive interruption handling, distinguishing true user cut-ins from background noise or brief pauses, and integrates with leading STT/TTS providers.
• AssemblyAI Universal-Streaming : Ideal if you prefer a modular STT-first approach. It features intelligent, low-latency endpointing and turn detection over WebSockets to help your custom LLM orchestration layer decide precisely when a user has finished speaking or is trying to interrupt.

If you can share whether you want a fully hosted speech-to-speech model (like OpenAI/Gemini) or a custom modular pipeline (using separate STT/LLM/TTS providers), I can help you choose the ideal architecture and protocol (WebSockets vs. WebRTC).