Speech-to-Speech vs Pipeline Voice Agents: Which Architecture in 2026?

There are two ways to build a voice AI agent in 2026, and the choice shapes everything downstream — latency, cost, how you debug it, and whether you can pass a compliance review.

The old way chains three models together. The new way uses one model that hears and speaks directly. Both are in production at scale. Neither is universally correct, and the honest answer to “which one” depends on constraints most comparison posts skip.

The two architectures

Cascading pipeline (STT → LLM → TTS)

Audio comes in, a speech-to-text model transcribes it, the text goes to an LLM, the LLM’s text reply goes to a text-to-speech model, audio goes out. Three models, three vendors if you want, and readable text between every stage.

This is the architecture nearly every voice agent used before 2025, and it’s still the majority of what’s running in production.

Speech-to-speech (S2S / realtime)

Audio goes into a single multimodal model and audio comes out. No text step in the middle. OpenAI’s Realtime API and Google’s Gemini Live native audio are the two main options.

Because the model hears the actual audio, it has access to information the pipeline throws away: tone, pace, hesitation, emphasis, laughter, frustration. And because it emits audio directly, its replies carry prosody that a TTS reading flat text can only approximate.

Speech-to-speech vs pipeline: the trade-offs

	Cascading pipeline	Speech-to-speech
Latency floor	Higher — three hops	Lower — one hop
Prosody & emotion	Lost at the STT step	Preserved end to end
Transcript	Native, at every stage	Available, but a side output
Guardrails	Deterministic, on text	Harder — you’re filtering audio intent
Tool / function calling	Mature	Supported, less battle-tested
Vendor swap	Any component, independently	Whole-stack lock-in
Model choice	Any LLM, including open or fine-tuned	Whatever the provider ships
Cost control	Fine-grained, per component	Coarse, per audio token
Debuggability	Read the text at each stage	Listen to the audio
Best for	Transactional, regulated, tool-heavy	Conversational, emotional, latency-critical

The latency question, honestly

This is where most comparisons go wrong. They quote the pipeline at 2–4 seconds and speech-to-speech at half a second, then declare a winner.

That 2–4 second figure is a non-streaming pipeline — one that waits for the caller to finish, waits for a full transcript, waits for the complete LLM response, then starts synthesis. It’s the naive implementation, and yes, it feels broken.

A streaming pipeline is a different machine. The realistic component budget:

Stage	Streaming budget
Speech-to-text (first partial)	60–100ms
LLM (first token)	100–180ms
Text-to-speech (first chunk)	40–80ms

That puts sub-250ms at p50 within reach, with a more typical production stack landing under ~700ms once you add network, turn detection and telephony overhead. Deepgram’s Nova-3 sits at the fast end of streaming STT, and Cartesia’s Sonic line reports time-to-first-byte in the tens of milliseconds.

On the other side, published speech-to-speech numbers are good but not instant either. Artificial Analysis measured Gemini 2.5 Flash native audio dialog at roughly 0.63s time to first audio, and OpenAI’s realtime models range from around 300ms up to a couple of seconds depending on how much reasoning is enabled.

The real conclusion: speech-to-speech has the lower floor, but architecture choice is not what separates a fast agent from a slow one. Streaming at every layer is. A streaming pipeline beats a badly integrated S2S deployment every time.

For reference, natural human conversation has turn-taking gaps around 200ms. Under ~500ms feels conversational; past a second, callers assume the agent didn’t hear them and start talking over it.

What speech-to-speech actually buys you

Not just speed. The underrated wins:

• It hears how something was said. “Fine.” delivered flatly and “Fine!” delivered brightly are the same six characters to an STT model. An S2S model hears the difference and can respond to it.
• Interruptions feel human. Barge-in and backchannels (“mhm”, “right”) are native to the audio stream rather than something you bolt on with voice activity detection.
• Fewer moving parts. One vendor, one connection, one failure mode instead of three.

If your product is companionship, coaching, language practice, or anything where how the agent sounds is the value, these are not nice-to-haves. They’re the product.

What the pipeline actually buys you

Also not just cost. The underrated wins:

• A text transcript at every stage, for free. Not an afterthought — the native data format. That’s your logging, your evals, your QA sampling, your compliance record, your fine-tuning dataset.
• Deterministic guardrails. You can regex, classify and validate text before it reaches TTS. Blocking a phrase in an audio stream is meaningfully harder.
• Any model you want. A fine-tuned small model, an open-weight model on your own hardware, a domain-specific one. S2S limits you to what the provider offers.
• Independent optimization. Latency problem in TTS? Swap TTS. Cost problem in the LLM? Drop to a smaller one. You don’t renegotiate the whole stack.

If your agent books appointments, handles support tickets, qualifies leads, or touches anything regulated — this list is why most production phone agents are still pipelines.

How to choose: a straight decision guide

Choose a cascading pipeline if:

• You need call recordings, transcripts or audit logs (support, healthcare, finance, collections)
• The agent does real work through tools — lookups, bookings, CRM writes
• You need to control cost per minute tightly at volume
• You want to fine-tune on your own conversation data
• You need a specific LLM the S2S providers don’t offer

Choose speech-to-speech if:

• Conversational naturalness is the product
• Emotional nuance matters — coaching, therapy-adjacent, companionship, language learning
• You’re latency-bound and every 100ms shows up in your metrics
• The tool surface is small and simple
• You want to ship a prototype fast with one vendor

A note on hybrids: the split isn’t permanent. Some teams run S2S for the conversational layer and route tool-heavy turns through a text path, or run a parallel STT stream purely for transcripts while S2S handles the conversation. That gets you auditability without giving up prosody — at the cost of a more complex system and paying for transcription twice.

The practical starting point

Whichever way you lean, don’t hand-roll the plumbing. Pipecat — open-source Python from Daily, BSD-2, v1.0 since April 2026 — models the agent as a pipeline of frame processors and ships integrations for most STT, LLM and TTS vendors, plus the realtime S2S APIs. Swapping a provider is a config change. LiveKit Agents (Apache-2.0) is the other serious option, and gets you a battle-tested WebRTC media layer underneath.

The reason this matters for the architecture decision: with a framework like Pipecat, you don’t have to be right on day one. Build on the pipeline, measure real latency and real cost with real users, and switch the conversational layer to S2S if the numbers justify it.

The things that will actually break your agent

Neither architecture saves you from the problems that decide whether a voice agent survives contact with users:

1. Turn detection. Knowing when the caller has finished speaking versus paused mid-thought. Get this wrong and the agent interrupts constantly — users forgive latency far sooner than they forgive being cut off.
2. Barge-in handling. When the user talks over the agent, audio output has to stop immediately and the partial response has to be discarded from history cleanly.
3. Telephony reality. Phone audio is narrowband and adds its own latency on top of your budget. A stack that feels great over WebRTC in the browser can feel sluggish on an actual PSTN call.
4. Context growth over long calls. A twenty-minute call accumulates a lot of history, and the same context engineering problems that hit text agents hit voice agents — with a latency penalty attached to every extra token.
5. Graceful failure. Providers deprecate models and APIs time out. A fallback chain isn’t optional; it’s the difference between a degraded call and a dead one.

The bottom line

Speech-to-speech is genuinely better at sounding human, and that gap is not closing from the pipeline side — text throws the information away and it can’t be recovered.

Cascading pipelines are genuinely better at being operated — logged, audited, guardrailed, cost-tuned and vendor-swapped — and that gap isn’t closing quickly either.

Most teams should start with a streaming pipeline, get turn detection right, measure, and move the conversational layer to speech-to-speech only when they can name the metric it improves. Architecture is a smaller lever than execution: a well-streamed pipeline beats a poorly integrated realtime model, every time.

---

I build production voice agents — Pipecat pipelines, telephony integration, and the caller-memory layer underneath — from Kotli, Azad Kashmir. If you’re deciding between these two architectures for a real deployment, see what I do or look at the projects.