Real-Time AI Voice Infrastructure: WebRTC, LiveKit, and Building the Conversational AI Backend That Doesn't Sound Like a Phone Tree

I have spent the last year and a half helping teams wire up conversational AI backends, and the pattern I keep seeing is the same one I watched play out with microservices eight years ago: engineers grab the obvious tool (an HTTP REST API), get something working in a demo, push it toward production, and then discover why that approach doesn’t hold up when a real human is on the other end of the line. The voice AI space is full of that right now.

The problem is not the LLMs. Models are extraordinary. The problem is the infrastructure layer between the human’s mouth and the model’s brain, and between the model’s output and the human’s ear. In my twenty years of building distributed systems, I have never worked on anything where the feedback loop is this tight. A database query that takes 200ms is fine. A voice response that takes 700ms sounds broken. The tolerance for latency in conversational AI is measured in dozens of milliseconds, not hundreds, and that constraint ripples through every architectural decision you make.

This article is about what it actually takes to build that infrastructure: WebRTC, LiveKit, the STT-to-LLM-to-TTS pipeline, how to run it on Kubernetes, how to think about cost, and the production lessons that only show up after you have shipped something and had real users talk to it.

Why WebRTC, Not HTTP

The first instinct most teams have is to build voice AI the way they build everything else: the client records audio, sends it as a POST request, the server returns audio as a response, the client plays it back. I have watched this work beautifully in demos and fall apart immediately in production.

HTTP round-trips are designed for request-response patterns where latency is measured in hundreds of milliseconds and a few percent jitter does not matter. Real-time audio is the opposite: it requires a continuous bidirectional stream, it requires sub-50ms network jitter, it requires the client to be able to interrupt mid-response (more on that shortly), and it requires audio to keep flowing even when packet loss happens. HTTP, even over HTTP/2, is the wrong abstraction.

WebRTC was designed for exactly this. It runs over UDP with the SRTP protocol on top for encryption. It has built-in jitter buffers, adaptive bitrate, packet loss concealment, and echo cancellation. It handles the session negotiation (SDP, ICE candidates, STUN/TURN) that lets two endpoints find each other across NAT and firewalls. And critically, it keeps a persistent connection open, which means you can stream audio bidirectionally in real time with a consistent ~20ms packet cadence.

For voice AI in 2026, WebRTC is not an option you choose. It is the substrate. Every serious voice AI infrastructure layer runs on it.

LiveKit: The De Facto Standard

If WebRTC is the substrate, LiveKit is what you actually build on. LiveKit is an open-source Selective Forwarding Unit (SFU) and real-time communications framework that has become the infrastructure of choice for voice AI workloads. The reasons are practical.

First, running a WebRTC SFU correctly is hard. You need to handle STUN/TURN for NAT traversal, manage ICE restarts when network conditions change, implement bandwidth estimation, and handle the signaling server that coordinates connection setup. LiveKit handles all of that.

Second, LiveKit’s Agents framework treats the AI as a full WebRTC participant in the room, not an external service you pipe audio to. The agent joins the room, subscribes to the user’s audio track, processes it, and publishes its own audio track back. This means interruption handling, multi-speaker scenarios, and participant lifecycle events are all first-class concerns built into the framework, not bolted on afterward.

Third, the ecosystem around LiveKit has consolidated. Deepgram, ElevenLabs, Cartesia, and OpenAI all ship official LiveKit plugins. When you are debugging a latency issue at 2 AM, the last thing you want is to be maintaining your own WebRTC integration from scratch.

You have two deployment choices: LiveKit Cloud (managed, $50/month for up to 1,000 concurrent participants) or self-hosted on Kubernetes. For teams with compliance requirements or cost sensitivity at scale, self-hosted is the right answer. For everyone else, start with LiveKit Cloud and migrate if you hit the limits.

The Latency Budget: 300ms Is the Ceiling

Before you design anything, internalize this number: 300ms. That is the maximum time from the end of a user’s utterance to the first syllable of the agent’s response before the conversation starts feeling unnatural. Under 200ms and it feels like talking to a sharp human. 200-300ms and users accept it. Beyond 300ms and they start wondering if the call dropped.

That 300ms has to cover the entire round trip: Voice Activity Detection (VAD) to detect the end of turn, Automatic Speech Recognition (STT) to transcribe the audio, LLM inference to generate the response, text-to-speech synthesis (TTS) to generate the first audio chunk, and the time for that chunk to reach the user’s ear. Here is what a realistic budget looks like for each component in 2026:

• VAD (end-of-turn detection): 50-100ms. This is frequently the biggest hidden cost. Bad VAD adds latency by not detecting the end of speech quickly enough, or creates false positives by cutting the user off mid-sentence.
• STT (first token latency): 150-250ms for streaming transcription with services like Deepgram Nova-3. You are not waiting for the full transcript. You start sending audio to the LLM as soon as you have a high-confidence partial transcript, overlapping the inference window.
• LLM (time to first token): 100-300ms for a fast model like GPT-4o-mini or Gemini 2.0 Flash, 200-500ms for a larger model. This is where model selection matters enormously. You cannot use a high-latency reasoning model for voice.
• TTS (time to first audio chunk): 100-250ms for streaming TTS with Cartesia Sonic or ElevenLabs Turbo v2. Again, you do not wait for the full response. You start playing audio the moment the first chunk arrives.
• Network and jitter: 20-60ms each leg if you have deployed your infrastructure close to your users.

The math reveals the challenge immediately. Each component has to be close to its lower bound for the end-to-end experience to hit 300ms. If your LLM is running on an overloaded GPU cluster and time-to-first-token spikes to 600ms, your voice agent feels broken regardless of how good everything else is. Latency in this pipeline is the maximum, not the sum.

Pipeline Architecture: Cascade vs. Native Multimodal

There are two fundamental pipeline architectures for voice AI in 2026, and the choice between them has significant infrastructure implications.

The cascade pipeline is STT-LLM-TTS: the user speaks, you transcribe with a dedicated speech recognition model, pass the text to an LLM, and synthesize the response with a dedicated TTS service. The vast majority of production voice AI deployments run this architecture. Its advantage is modularity: you can swap components independently, you can use the best STT for your language and use case, the best LLM for your domain, and the best TTS voice for your brand. Its disadvantage is that each handoff adds latency and loses information. The LLM never hears the user’s tone, hesitation, or emotional state. It only gets text.

The native multimodal pipeline sends audio directly to a model that processes it end-to-end: OpenAI’s Realtime API with GPT-4o Audio, or Google’s Gemini 2.0 Flash native audio mode. These models hear the audio directly, understand prosody and affect, and output audio without an intermediate transcription step. The latency can be lower (130-200ms to first audio chunk in good conditions), the conversation feels more natural, and you get richer emotional understanding. The tradeoff is that you are locked into a single vendor for the entire pipeline, the per-minute costs are higher, and reliability depends entirely on that provider’s uptime.

My recommendation for most teams in 2026: start with the cascade pattern. It is easier to debug, cheaper at scale, and gives you the flexibility to optimize each component independently. Migrate to native multimodal for specific use cases where affect understanding or absolute minimum latency is worth the tradeoff. This aligns with how I think about LLM inference more broadly: right model for the right task, not the newest or most impressive model by default. The inference engine selection decision matters as much for voice as it does for any other AI workload.

Turn Detection and Interruption Handling

This is where most voice AI projects fall apart in production, and it is a problem that no amount of cloud infrastructure can solve if the logic is wrong.

Turn detection has two failure modes. False negatives mean the agent waits too long to respond because it does not realize the user has finished speaking, killing the feel of natural conversation. False positives mean the agent starts responding while the user is still mid-sentence, talking over them. Both are worse than a slow response because they feel like rudeness, not latency.

The VAD (Voice Activity Detection) model is responsible for turn detection. LiveKit’s built-in VAD uses Silero VAD by default, which is competent but has fixed silence thresholds that do not adapt to context. Better approaches use a combination of acoustic VAD (is the user still making sounds?) and semantic VAD (has the user completed a thought?). The semantic layer passes the in-progress partial transcript to a lightweight model to predict whether the sentence is likely complete. This adds a few milliseconds of latency but dramatically reduces the false positive rate.

Barge-in handling, where the user interrupts the agent mid-response, is the other hard problem. The agent is speaking. The user says something. The agent should stop immediately and respond to the new input. This sounds simple and is genuinely complex in practice. The agent’s audio is playing through the user’s speakers, which means the user’s microphone is picking up the agent’s audio (acoustic echo cancellation handles this at the WebRTC layer, but imperfectly). The agent needs to detect that the user is speaking, cancel its own TTS playback, discard any buffered LLM output, and restart the pipeline from the new user input. Doing this without race conditions, without double-processing the echo, and without audible glitches requires careful sequencing.

LiveKit’s agent framework gives you the primitives to implement this, but the logic is yours to write. I have seen teams spend more engineering time on barge-in handling than on the entire initial pipeline integration, which is worth knowing before you estimate.

Kubernetes Deployment Architecture

For self-hosted LiveKit and AI agent workers, Kubernetes is the right platform. Here is the architecture I use.

LiveKit Server itself is stateless (session state is stored in Redis). Deploy it as a Deployment with a minimum of 3 replicas spread across availability zones. Expose it with a LoadBalancer service or, if you need geographic routing, behind a Global Load Balancer. LiveKit uses both TCP (WebSocket for signaling, port 7880) and UDP (RTP/SRTP for media, ports 50000-60000 by default). The UDP port range is the tricky part: you need to punch that range through your cloud security groups and ensure your Kubernetes nodes have routable external IPs or you have a TURN server for NAT traversal.

The AI agent workers are where GPU scheduling matters. Each voice session spawns an agent process that subscribes to the LiveKit room and runs the STT-LLM-TTS pipeline. If you are running local STT (Whisper) or local TTS inference on-device rather than hitting cloud APIs, those processes need GPU access. This is where Kubernetes pod scheduling with taints and tolerations becomes important: you want your voice agent pods to land on GPU nodes, and you want to prevent non-GPU workloads from competing for those resources.

A typical production setup:

• LiveKit Server: CPU nodes, 2-4 vCPUs, 8GB RAM per replica
• Agent workers (cloud STT/TTS): CPU nodes, 2 vCPUs, 4GB RAM, scale with Kubernetes HPA based on active session count
• Agent workers (local inference): GPU nodes (A10G or H100 depending on model size), scale with KEDA based on LiveKit room occupancy
• Redis: managed Redis cluster for LiveKit session state (ElastiCache, Upstash, or similar)

One thing I learned building this: do not try to multiplex multiple voice sessions onto a single agent process unless you have built very careful isolation between them. The temptation to pack sessions onto a single GPU for efficiency is real, especially when your GPU costs are high. But a single slow LLM call in one session can block all sessions running in that process, and the resulting latency spike is catastrophic for voice.

STT, LLM, and TTS: Vendor Selection for Production

The component choices matter more in voice AI than in most AI applications because you are latency-constrained at every stage.

For STT, Deepgram Nova-3 is the production standard for English in 2026: streaming partial transcripts in near-real-time, solid on telephony audio (8kHz, noisy), sub-200ms API latency for the first transcript. For on-prem or privacy-sensitive deployments, Whisper Large v3 Turbo via a dedicated inference server works, at slightly higher latency.

For LLMs in a voice pipeline, optimize for time-to-first-token, not throughput. Reasoning models designed for batch processing are wrong for voice. GPT-4o-mini, Gemini 2.0 Flash, and Claude Haiku 4.5 hit the right speed-to-capability balance for most use cases. The agentic AI scaling patterns I have used for text agents translate directly here.

For TTS, Cartesia Sonic and ElevenLabs Turbo v2.5 both begin streaming audio within 80-150ms of receiving text. Cartesia has a slight latency edge; ElevenLabs has a larger voice library. For on-prem TTS, Kokoro on a GPU gives acceptable quality at significantly lower cost, with 30-50ms additional latency.

Cost Architecture

Voice AI is expensive relative to text AI, and the cost model is different enough to trip up teams that are used to thinking in token budgets.

The primary cost driver is the per-minute pricing for streaming STT and TTS. Deepgram charges around $0.01/minute for STT. ElevenLabs Turbo v2.5 is around $0.03-0.05/minute for TTS. LLM costs are based on tokens as usual, but voice conversations tend to generate significantly more tokens per minute than you expect because of the turn-by-turn format (every user utterance gets transcribed, every agent response gets generated, and conversations tend to be longer than text chats).

A rough cost estimate for a 10-minute voice call using cloud services:

• STT: ~$0.10
• LLM (GPT-4o-mini, ~2,000 tokens/minute average): ~$0.04
• TTS: ~$0.40
• LiveKit session (if using Cloud): $0.01
• Total: ~$0.55 per 10-minute call

For high-volume deployments, that $0.55 per call gets meaningful fast. The optimization strategies I’ve seen work at scale:

First, aggressive turn caching. If you have a customer service agent that answers the same 50 questions 90% of the time, cache the TTS audio for common responses. The first caller gets the fresh synthesis; subsequent callers get the cached audio file. This is not possible for fully dynamic conversations but it applies more often than teams expect. This is the audio equivalent of prompt caching for LLMs.

Second, self-hosted TTS for high-volume. Running Kokoro or a fine-tuned Coqui model on an A10G GPU reduces TTS cost by roughly 80% at scale. The quality gap versus ElevenLabs is real but narrow for professional use cases.

Third, right-sizing the LLM. Most voice use cases do not need GPT-4o. For customer service, appointment scheduling, FAQ bots, and similar high-volume narrow-domain tasks, a fine-tuned smaller model often outperforms the general-purpose large model while costing 10-20x less per token. Explore serverless GPU platforms like Modal for running these fine-tuned models without managing GPU infrastructure permanently.

Observability for Voice AI

Standard APM tooling does not capture what matters for voice AI. You care about per-session metrics that most observability systems are not designed to track.

The metrics that matter:

• Time to First Utterance (TTFU): End of user speech to first audio byte of response. This is your primary KPI.
• Interruption rate: How often the user interrupts the agent. High interruption rate means your turn detection is too slow or your responses are too long.
• False trigger rate: How often the agent starts responding when the user is still speaking. Measure this with post-processing on transcripts.
• STT confidence distribution: Low-confidence transcriptions lead to wrong LLM inputs and bad responses. Track the 5th percentile confidence.
• Session abandonment: Users who disconnect within 30 seconds are usually experiencing an unacceptable latency or quality issue.

Implement OpenTelemetry spans for each pipeline stage with TTFU broken down by component. The breakdown tells you immediately whether your STT, LLM, or TTS is the current bottleneck. LiveKit emits WebRTC quality metrics (jitter, packet loss, RTT) natively; ingest those into your observability pipeline alongside your application metrics.

One instrumentation pattern I’ve found invaluable: record audio of every session (with user consent and appropriate data handling) and build an automated quality scoring pipeline that transcribes the agent’s responses and evaluates them against your quality rubric. You discover things in the aggregate that you would never catch listening to individual sessions.

Production War Stories

Two failure modes I have encountered are worth calling out because they are not obvious from documentation.

The “acoustic echo” false trigger problem. In a deployment where users called from speakerphone, the agent’s TTS audio played through their speakers and was picked up by their microphone, triggering the VAD. WebRTC’s acoustic echo cancellation (AEC) is supposed to handle this, but it requires accurate echo delay estimation. When you inject audio into a WebRTC session programmatically rather than through a physical speaker, the echo delay differs from what the AEC model expects. The fix was LiveKit echo cancellation settings tuned for agent audio injection, combined with higher VAD confidence thresholds during active TTS playback.

The LLM streaming backpressure problem. The LLM streamed tokens fast enough, but TTS occasionally fell behind, creating a growing buffer of unsynthesized text. When the user interrupted, we cancelled LLM generation and the TTS queue, but had not implemented backpressure signaling from TTS back to LLM. The result: a fragment of the cancelled response played after the interruption before the new response started. The fix was a cancellation token propagating through the entire pipeline, with a TTS chunk buffer drain check before the pipeline accepted new input.

Build testing infrastructure that simulates realistic network conditions, device types (speakerphone behaves very differently from headsets), and interruption patterns. The AI agent orchestration patterns I have borrowed from LangGraph have helped manage pipeline state correctly around cancellation.

Architecture Decision: Should You Build or Buy?

The voice AI space has a growing set of fully managed platforms that handle the entire stack: Vapi.ai, Bland.ai, and Retell AI among them. These give you a voice agent in a few API calls without thinking about WebRTC or SFUs or VAD.

The build-on-LiveKit approach makes sense when:

• You have compliance requirements that prevent third-party audio processing
• You need deep customization of the pipeline (custom VAD, specialized STT models, domain-specific TTS voices)
• You are at a volume where the per-minute API costs of managed platforms exceed the engineering cost of building and maintaining your own infrastructure
• You need to run entirely on-premises or in your VPC

The fully managed platform approach makes sense when:

• You are building a proof of concept or early product and iteration speed matters more than cost or customization
• Your team does not have WebRTC expertise and you do not want to acquire it
• Your use case is standard enough that the platform’s defaults work for you

In my experience, teams that build on managed platforms often migrate to self-hosted LiveKit once they hit production scale. The economics of $0.50+ per call become compelling to optimize once you are doing tens of thousands of calls per day.

Where This Goes

Two trends I am watching closely.

First, hardware acceleration for voice pipelines. The latency bottlenecks today are largely in software: STT and TTS inference on CPUs or underutilized GPUs. Purpose-built silicon for speech processing is likely to emerge in the next cycle, potentially cutting pipeline latency in half.

Second, multimodal native models eating the cascade pipeline. As GPT-4o audio and Gemini native multimodal improve, the appeal of a three-component pipeline weakens. When a single model can hear audio, understand prosody, and respond in audio with sub-150ms latency, STT-LLM-TTS becomes legacy architecture. I expect the cascade pattern to remain dominant for the next 18 months while native multimodal latency continues to drop. When it does shift, it will shift quickly.

The infrastructure changes with native multimodal too. Instead of managing three services, you manage one high-capability model with very strict latency SLAs. The multi-region active-active deployment patterns become critical for always routing to a model endpoint that can hit your latency budget.

Keep the pipeline modular, invest in observability, and do not over-optimize for a specific vendor’s API at the expense of flexibility. The voice AI platform you run in two years will share the same WebRTC and Kubernetes substrate you build today, but almost nothing else will look the same.