I'm looking for an API that provides real-time, accurate speaker diarization (who spoke when) from an audio stream.

If you need real-time speaker diarization (“who spoke when”) from an audio stream, the best API depends on whether you need live captions + speaker labels, raw diarization segments, voice identity, or meeting-grade separation. These are some of the strongest options:

Strong API choices

1. deepgram.com — best general-purpose streaming option

Deepgram supports streaming transcription with diarization enabled. Its streaming output can attach speaker IDs to words, allowing you to reconstruct segments like:

{
  "speaker": 0,
  "word": "hello",
  "start": 1.24,
  "end": 1.52
}

It is a good fit for:

• call centers
• live captions
• voice assistants
• interviews

Deepgram documents streaming diarization through its diarize_model configuration.

---

2. assemblyai.com — strong conversational diarization

AssemblyAI’s Streaming API supports real-time speaker labeling. It can emit turn-level labels (speaker_label) and word-level speaker information for detecting changes inside a turn.

Good for:

• customer conversations
• meeting transcription
• conversational analytics

---

3. platform.openai.com Realtime API — best if diarization is part of a voice agent

OpenAI’s realtime audio stack supports transcription models with diarization capabilities, including gpt-4o-transcribe-diarize.

Good for:

• AI phone agents
• interactive assistants
• apps where you also need reasoning/dialog management

---

4. aws.amazon.com — best AWS-integrated choice

Amazon Transcribe supports real-time streaming speaker partitioning and speaker labels.

Good for:

• enterprises already on AWS
• compliance-heavy environments
• existing Kinesis/Connect pipelines

---

If you need true “who is this person?” identification

Most diarization APIs only answer:

Speaker A → Speaker B → Speaker A They do not inherently know that Speaker A is “Alice.”

For identity matching you need:

• speaker enrollment (voiceprints)
• speaker recognition
• a voice biometric system

That is a different problem from diarization.

---

My shortlist by use case

• Lowest latency live transcript + speakers: Deepgram or AssemblyAI
• AI voice assistant: OpenAI Realtime API
• Enterprise AWS stack: Amazon Transcribe
• Zoom/Teams/Meet ingestion: Recall.ai
• Privacy/self-hosting: pyannote/diart-style pipelines

A few questions would narrow this down:

1. Do you need raw speaker timestamps only, or transcription + speaker labels?
2. How low does latency need to be (e.g., <500 ms, <2 s)?
3. How many simultaneous speakers (2-person call vs meetings with 10+ people)?
4. Do you need to identify speakers by name, or just label them A/B/C?

AssemblyAI, Deepgram , and Speechmatics offer the most robust, highly accurate APIs for real-time speaker diarization from a live audio stream. Implementing speaker diarization over a live stream is technically challenging because models typically need context from the entire audio file to group speakers accurately. However, several top-tier providers offer this via WebSocket connections for low-latency streaming.

Top Real-Time Speaker Diarization APIs

Deep Dive: How Leading APIs Handle It

1. AssemblyAI (Universal-3.5 Pro Realtime)

AssemblyAI is a premier choice for developer-friendly integration. They offer Streaming Speaker Diarization directly through their real-time WebSocket connection.

• How it works: You pass speaker_labels: true in your initial WebSocket configuration. As you stream binary audio blocks, the API returns JSON objects containing the transcription text and a stable speaker channel ID (e.g., Speaker 1, Speaker 2).
• The Advantage: Their "Context Carryover" ensures that even if a speaker stops talking and starts again later, the system remembers their voice profile and retains the same ID.

2. Deepgram Nova-2 Streaming

Deepgram is known for processing speed and high-volume scalability.

• How it works: By enabling diarize=true in your streaming query parameters, Deepgram's deep learning model analyzes word-level timestamps.
• The Advantage: It returns results containing an assigned speaker integer for every single word token. It is optimized to handle overlapping talkers and heavy background noise effortlessly.

3. PyAnnote.ai (For Open Source & Custom Control)

If you want to avoid strict vendor lock-in, PyAnnote is the absolute gold standard open-source framework for diarization.

• How it works: While historically used for self-hosted batch files via Hugging Face, they now offer pyannoteAI , a managed pipeline that delivers real-time speaker insights in under 150 ms.

Important Developer Trade-offs to Consider

• Correction and Revision: In real-time streaming, you will occasionally see a speaker ID change retroactively. APIs send "interim" results quickly, followed by a "final" block once the model has a few extra milliseconds of audio context to confirm who spoke. Your UI layer must be built to update these retroactively.
• Multi-Channel vs. Single-Channel: If you can capture separate audio tracks (e.g., from separate microphones or distinct browser audio streams), skip AI diarization entirely and use multi-channel transcription. It guarantees 100% accurate speaker sorting. Use AI diarization only when you are forced to process a single, mixed audio stream.

To help narrow down the best choice, what is your primary programming language , and what type of application (e.g., a phone system, a web meeting app, or a smart device) are you building?

For real-time, streaming speaker diarization (identifying "who spoke when" over a live audio feed via WebSockets), a few top-tier developer APIs stand out for accuracy, low latency, and ease of implementation.

Top API Recommendations

• AssemblyAI — Highly regarded for developer-friendly real-time transcription and streaming speaker labels. You can opt-in to diarization directly on a live WebSocket connection by passing speaker_labels: true . It streams word-level speaker attribution and turn events through the same socket. Check out the technical details on the AssemblyAI Streaming WebSocket API Documentation.
• Deepgram — Offers lightning-fast streaming Speech-to-Text with support for real-time speaker identification and diarization models (diarize_model ). Deepgram is heavily optimized for low-latency voice agents and contact centers. Explore how to configure it via the Deepgram Diarization Documentation.
• Gladia — Known for ultra-low latency audio intelligence (<300ms) with support for live, real-time multilingual speaker diarization and rich audio metadata over streaming endpoints. Learn more on the Gladia Platform.

How Real-Time Streaming Diarization Works

1. Connection: You open a persistent WebSocket connection to the provider's streaming endpoint.
2. Payload: You stream raw audio chunks (e.g., PCM, Opus) from your client or server.
3. In-Stream Tagging: Instead of a separate batch post-processing call, the incoming websocket frames return incremental transcript turns bundled with a speaker or speaker_label field dynamically as the conversation progresses.

If you share details on your setup, I can help tailor this further:

• Are you streaming from a browser/client-side or a server-side backend?
• What audio format or codec are you working with?
• Do you have an expected maximum number of simultaneous speakers?

Yes. If your requirement is specifically real-time speaker diarization from a live audio stream, I’d shortlist these:

My recommendation: AssemblyAI

AssemblyAI now supports real-time diarization directly on its streaming WebSocket. You enable speaker_labels: true, and each live turn gets a speaker label; it supports up to 10 speakers and lets you provide max_speakers when you know the expected count.

The particularly interesting part is SpeakerRevision: as of June 2026, AssemblyAI can revise speaker assignments using additional context at the end of the stream, rather than leaving you with potentially incorrect early labels. They report roughly 400 ms average additional latency for the revised result.

For an application where "accurate" matters more than merely getting a speaker ID quickly, that's a significant advantage.

Deepgram

Deepgram is probably the strongest alternative if you're already using its transcription stack. Its streaming API supports diarization and returns a speaker value associated with words. The current streaming diarizer is v1; latest resolves to that streaming model.

One important distinction: live diarization isn't the same problem as batch diarization. A streaming system has to assign a speaker before it has heard the entire conversation, so early labels can be less stable. AssemblyAI's newer revision mechanism specifically addresses that limitation.

What I'd choose by use case

• 2–4 people in meetings/calls: AssemblyAI
• Voice agent / live conversation: AssemblyAI
• Existing Deepgram infrastructure: Deepgram
• Many languages: benchmark AssemblyAI and Soniox on your actual recordings
• Need final transcript to be extremely accurate: AssemblyAI with SpeakerRevision
• Need sub-second live speaker attribution: test both AssemblyAI and Deepgram with your actual audio rather than relying on published benchmarks

One caveat: no streaming diarization API is perfectly accurate, especially with overlapping speech, far-field microphones, background noise, or very short interjections. For example, AssemblyAI explicitly notes that overlapping speech can be problematic when everyone is coming through one channel.

If you tell me (1) number of speakers, (2) audio source—phone call, microphone, Zoom/WebRTC, etc., (3) languages, and (4) your maximum acceptable latency, I can narrow this to the best 2–3 APIs and compare accuracy, latency, pricing, WebSocket protocol, and sample code.

For an application where "accurate" matters more than merely getting a speaker ID quickly, that's a significant advantage.

Deepgram

Deepgram is probably the strongest alternative if you're already using its transcription stack. Its streaming API supports diarization and returns a speaker value associated with words. The current streaming diarizer is v1; latest resolves to that streaming model.

Yes. If your requirement is real-time speaker diarization from a live audio stream—i.e. receiving events like Speaker 0 started → Speaker 1 → Speaker 0 while the audio is still arriving—there are several viable APIs.

My shortlist

1. Deepgram — probably my first choice

Deepgram explicitly supports diarization on live streaming requests. Its current streaming diarizer is v1; diarize_model=latest resolves to the latest streaming diarizer. The API assigns speakers to words, which makes it possible to construct your own utterance-level events.

Conceptually, you'd get output like:

00:00.0  speaker_0  "Hello, thanks for calling."
00:02.4  speaker_1  "Hi, I have a question about my order."
00:05.1  speaker_0  "Sure, what's the order number?"

For an application where latency matters, this is the one I'd benchmark first.

2. Amazon Transcribe

AWS supports speaker partitioning directly in streaming transcription. You can use HTTP/2 or WebSockets and enable show-speaker-label. AWS says standard Transcribe can distinguish up to 30 speakers, although the streaming documentation notes that partitioning works best with roughly 2–5 speakers.

This is particularly attractive if you're already running your audio infrastructure on AWS.

One important distinction

There are actually two different problems people call "real-time diarization":

1. Streaming speaker labeling:
   "speaker_0 said these words, then speaker_1 said these words" as the stream progresses.

2. True low-latency speaker-change detection:
   "speaker changed at precisely 1.83 seconds" with minimal delay, including overlapping speech.

Most cloud STT APIs are primarily designed around #1. They may revise partial results and only give you definitive speaker labels once a speech segment is finalized. AWS, for example, explicitly notes that speaker labels appear on fully transcribed segments in its streaming output.

If you need sub-second speaker-change detection, especially with overlapping speakers, interruptions, meetings, or noisy environments, I'd benchmark the APIs rather than choosing solely from published accuracy claims.

What I'd test

For your use case, I'd run the same ~30–60 minute corpus through Deepgram + AssemblyAI + Speechmatics + AWS, measuring:

• speaker attribution accuracy (DER/JER)
• speaker-change latency
• word-level timestamp accuracy
• overlapping speech
• cross-talk/interruption handling
• 2 vs. 5 vs. 10+ speakers
• microphone vs. phone-call audio
• CPU/network overhead
• cost per audio hour

If you tell me what kind of stream you have (WebRTC microphone, phone call/Twilio, Zoom/Meet, raw RTP, etc.), expected number of speakers, and your maximum acceptable latency (e.g. <300 ms, <1 sec, <3 sec), I can narrow this down to the 2–3 best APIs and compare their current pricing, latency, SDKs, and diarization accuracy.

Finding a reliable API for real-time streaming speaker diarization can be tricky because live streaming forces models to make attribution decisions with zero future context. However, several top-tier Speech-to-Text (STT) and Voice AI platforms support live WebSocket streaming with speaker labels.

The leading options for real-time streaming diarization include:

• AssemblyAI : Offers Universal-3.5 Pro Realtime , which supports streaming speaker diarization over WebSockets. You pass speaker_labels: true in your connection parameters, and it tracks up to 10 speakers live. It also utilizes diarization with revision , meaning it provides ultra-low latency live tags and quickly adjusts any early misattributions as the contextual audio window expands.
• Speechmatics : Provides real-time transcription with robust live speaker separation and tracking. By enabling their real-time diarization config, every word and punctuation object returned over their streaming channel includes a distinct speaker property (e.g., S1, S2 ). They excel at handling diverse acoustic environments and accents in a live stream.
• Soniox : Known for high-accuracy real-time speech recognition and automatic multi-speaker tracking over live streams. While real-time constraints inherently carry a slightly higher error rate than batch processing, Soniox handles rapid turn-taking and multi-language live contexts efficiently.
• Pyannote.ai : Offers a specialized streaming diarization service (Live-1 ) designed to run alongside standard streaming ASR pipelines. You route your audio stream to both your ASR provider and Pyannote's live endpoint, then reconcile the word timestamps with the live speaker events to build a real-time attributed transcript.
• Picovoice (Bluebird) : If your use case demands strict data privacy or edge execution, Picovoice offers Bluebird Streaming Diarization , which processes audio entirely on-device rather than sending it to a cloud API. It outputs speaker labels in under 250 ms with no speaker caps or network dependencies.

To help narrow down which of these fits your stack best, tell me:

• What is your target deployment environment (cloud API vs. edge/on-device)?
• What is the expected number of speakers per stream?
• Are you looking for a unified STT + Diarization API or just a standalone diarization router?

Yes. If real-time speaker diarization is the key requirement—not just transcription with speaker labels added after the fact—there are a few good options. As of August 2026, I’d shortlist these:

1. My first choice: pyannoteAI Live-1

This is the most purpose-built option. Live-1 is specifically designed to take an audio stream and continuously emit speaker events rather than trying to retrofit batch diarization into a live pipeline. It uses WebSockets, accepts 16-kHz mono audio in ~100 ms chunks, supports up to 8 speakers, and reports speaker start/end times and labels with sub-300-ms latency.

Most importantly, pyannote's published benchmark on DIHARD III showed a 19.8% DER for its streaming model versus 31.3% for Speechmatics, 39.1% for Deepgram, and 39.2% for AssemblyAI in that particular benchmark. Treat those numbers as directional rather than definitive—the benchmark was published by pyannote, so I'd validate against your own audio.

It also launched GA in July 2026, with published pricing starting at €0.198/hour on its developer plan.

2. Deepgram — easiest all-in-one option

Deepgram's streaming API supports diarization directly. You can enable diarize_model=latest, and each word in the streaming transcript receives a speaker number.

That's particularly attractive if your pipeline is:

microphone → WebSocket → transcription + speaker → LLM

rather than:

microphone → diarization API + separate STT → merge timestamps

One caveat: Deepgram's current streaming diarizer is v1; its newer v2 diarizer is not currently supported for streaming.

3. AssemblyAI

AssemblyAI also upgraded its streaming diarization substantially in May 2026. Its own published benchmarks report improvements in speaker attribution, including fewer phantom turns and misattributed words compared with Deepgram. Again, because these are vendor-produced benchmarks, I'd verify on your data.

One thing to watch: older AssemblyAI documentation describes streaming diarization primarily through multichannel audio, so make sure the current streaming API behavior matches your exact input architecture.

What I'd choose

If your requirement is literally:

"Here's a live mixed audio stream. Tell me immediately when Speaker A/B/C starts and stops talking, as accurately as possible."

I'd start with pyannoteAI Live-1.

If you need transcription + diarization from one provider, I'd benchmark Deepgram vs AssemblyAI alongside it.

If you tell me (1) expected number of speakers, (2) audio source—phone call, microphone, Zoom/WebRTC, etc., (3) whether you need transcription too, and (4) your maximum acceptable latency, I can narrow this to the best API and give you the actual WebSocket architecture/code.

For real-time, streaming speaker diarization, several cloud Speech-to-Text APIs support live speaker attribution via WebSockets . True real-time diarization is challenging because clustering voices accurately usually requires looking ahead at broader audio context, but modern production APIs handle this with low-latency streaming and dynamic label revisions.

The top API options for streaming speaker diarization include:

• AssemblyAI Universal-3.5 Pro Realtime – Offers state-of-the-art streaming diarization with continuous speaker tracking for up to 10 speakers. It outputs live speaker labels over WebSockets and applies a fast background revision within milliseconds of a phrase ending to correct early attribution errors.
• Deepgram Streaming Diarization – Provides high-throughput, low-latency WebSocket streaming. You can pass diarize_model=latest or diarize=true parameters in your live connection to receive continuous word-level speaker tags (speaker: 0, speaker: 1 ) alongside your transcript stream.
• Speechmatics Real-Time Diarization – Known for robust multi-speaker acoustic matching and precise speaker separation in live streams. When configured with diarization: "speaker" , each punctuation and word object returned in the real-time session includes an active speaker property (S1, S2 , etc.).
• pyannote.ai Live-1 – Specifically built around the industry-standard open-source diarization toolkit (pyannote.audio ), this specialized API offers dedicated block-online streaming diarization events optimized purely for identifying who is speaking in real time.

If you can share:

• Your target latency constraints (e.g., sub-300ms vs. a few seconds of look-ahead)
• The expected number of speakers or environment type (e.g., phone calls, multi-person meetings)

I can help you pick the best-fit API or provide a quick integration code snippet.

Yes. If your requirement is specifically real-time speaker diarization from a live audio stream, I’d shortlist these:

My first choice: pyannoteAI

If "accurate who-spoke-when" is the primary requirement, I'd start with pyannoteAI's new Live-1 streaming diarization. It accepts 16-kHz mono audio over WebSocket in 100-ms chunks, reports speaker labels with start/end times, supports up to 8 speakers, and advertises roughly 300 ms diarization latency.

There's also a particularly relevant independent-ish comparison published by pyannoteAI: on the DIHARD III benchmark, its streaming system reported 19.8% DER, versus 31.3% for Speechmatics, 39.1% for Deepgram Nova 3, and 39.2% for AssemblyAI. Treat that as a vendor benchmark rather than definitive proof—the important thing is to test your own audio.

Deepgram is the pragmatic alternative

If you need transcription + diarization in one streaming API, Deepgram is attractive. Its streaming API supports diarize_model=latest and returns speaker attribution alongside the transcript.

So your pipeline can essentially be:

Microphone / RTP / WebRTC
          ↓
     WebSocket
          ↓
   Deepgram / pyannoteAI
          ↓
{ speaker: 0, start: 12.4, end: 14.1, text: "..." }
{ speaker: 1, start: 14.2, end: 16.8, text: "..." }

One important distinction

If by "who" you mean anonymous speaker labels:

SPEAKER_00 → SPEAKER_01 → SPEAKER_00

then ordinary diarization is sufficient.

If you mean known identities:

Alice → Bob → Alice

you need speaker identification / enrollment in addition to diarization. pyannoteAI supports speaker identification as well as diarization.

Recommendation: For a product where diarization accuracy is genuinely mission-critical, I'd prototype pyannoteAI Live-1 vs. Deepgram on 1–2 hours of your actual audio and measure DER, speaker swaps, latency, overlapping speech, and late-joining speakers. Those measurements will be much more meaningful than generic API accuracy claims.

Several speech-to-text and AI infrastructure providers offer real-time streaming speaker diarization over WebSockets. Because analyzing multi-speaker streams on-the-fly requires balancing low latency with contextual clustering (remembering who "Speaker 1" was five minutes ago), performance varies depending on your requirements.

Top managed APIs and developer tools for real-time streaming speaker diarization include:

• AssemblyAI (via AssemblyAI Realtime API)
  • How it works: You connect via WebSocket, stream audio chunks, and set speaker_labels: true in your configuration. Their flagship Universal-3.5 Pro Realtime model supports up to 10 speakers per live session.
  • Key feature: It includes features like SpeakerRevision , which issues a refined structural pass shortly after a turn or stream ends to correct early speaker attribution errors without needing a separate batch pipeline.
• Speechmatics (via Speechmatics Real-time Diarization)
  • How it works: Real-time WebSocket architecture that supports multi-language streaming.
  • Key feature: Enabling diarization in the transcription_config assigns a speaker property (S1, S2 , etc.) to every incoming word and punctuation object dynamically in the live stream.
• Pyannote.ai (via pyannoteAI Live-1)
  • How it works: Built specifically around the industry-standard pyannote ecosystem for conversational intelligence. Live-1 runs alongside your streaming ASR provider, taking the same audio stream and outputting parallel speaker events that you reconcile by timestamp.
  • Key feature: Highly specialized in speaker tracking using an arrival-order cache, handling up to 8 speakers with sub-150ms to 250ms adjustments.
• Picovoice Bluebird (via Picovoice Streaming Diarization)
  • How it works: Unlike cloud-reliant alternatives, Bluebird processes streaming diarization entirely on-device (offline/private architecture).
  • Key feature: Zero network latency, no data leaves the platform, and there is no strict speaker cap or language restriction.

To help narrow down which of these fits best, tell me:

• Do you need cloud-based convenience or on-device/private processing?
• What is your expected maximum number of concurrent speakers in a single stream?