If you have built chat AI systems and are considering voice, the problems are not incrementally harder. Many of them are structurally different. I spent months building voice AI systems after years of working on chat, and the gap was bigger than I expected.
The Latency Spectrum
Email is the forgiving end. A response can take minutes. You can run complex retrieval pipelines and nobody notices.
Chat is in the middle. Users expect responses within a few seconds. Latency matters, but you have enough room to run a retrieval step, make a tool call, and generate a response without the experience feeling broken.
Voice is the extreme end. In a voice conversation, silence is failure. If the system takes two seconds to respond, the user assumes it is broken, starts talking again, and now you are handling overlapping speech while still generating the first response. The target we worked toward was 300 to 500 milliseconds from the end of user speech to the start of system speech. That includes everything: speech-to-text, processing, LLM generation, and text-to-speech.
This is not a performance optimization problem. The entire pipeline has to be designed around that latency constraint from day one.
Tool Calls Cannot Pause
In chat, a tool call that takes three seconds is invisible. The user sees a typing indicator and waits. In voice, three seconds of silence is devastating. The agent must actively manage the waiting: acknowledge that it is doing something, give a verbal filler, or provide a partial response while it works.
This changes how you design tool integrations. Every external call needs a latency budget, and if a tool might be slow, you need a fallback plan for what the agent says in the meantime.
Turn-Taking Is Unsolved
Knowing when the user has finished speaking is one of the hardest problems in voice AI. Humans are remarkably good at this in natural conversation. AI systems are not.
The system needs to distinguish between a pause because the user is thinking, a pause because they are done, a pause because they want acknowledgment, and the user talking over the system. In our testing, basic Voice Activity Detection misclassified turn boundaries roughly 15% to 20% of the time.
Modern approaches combine acoustic cues, prosody analysis, and semantic completeness. Transformer-based models like the Voice Activity Projection model can predict whether a speaker is about to continue or has finished. These cut the error rate significantly, but not enough to feel completely natural in every conversation.
Interruption handling is a related challenge. When a user speaks over the agent, the system needs to stop immediately, process the interruption, and respond to it.
Voice Quality Is a Product Decision
In chat, tone is a matter of word choice. You can adjust it in the system prompt. In voice, tone is literal. The voice itself, its accent, pace, pitch, and warmth, shapes how users perceive the product.
Choosing a voice is a product decision with real consequences. A voice that sounds too robotic undermines trust. A voice that sounds too human creates uncanny valley effects when the responses are wrong.
We evaluated several options for our pipeline. OpenAI and Google Gemini both offer strong speech synthesis. For transcription, NVIDIA's open-source Parakeet model is impressive, processing an hour of audio in roughly one second in batch mode on optimized hardware. Streaming latency is a different measurement, but the throughput gives you a sense of how fast these models have become. The right choice depends on your latency requirements, language support, and how much control you need over voice characteristics.
The Infrastructure Gap
A chat AI system is REST APIs and text processing. A voice AI system is a real-time multimedia pipeline.
Voice requires audio streaming over WebSockets, echo cancellation, noise suppression, careful buffer management, and coordination between multiple real-time processes. The speech-to-text step needs to run continuously on streaming audio, not in batch. The text-to-speech step needs to start generating audio before the full response is complete. Every component operates under strict timing constraints.
Most of this infrastructure does not exist in a chat system and cannot be retrofitted. When we moved from chat to voice, we underestimated the scope by a factor of three.
Why It Matters
Voice is worth building. But the path from chat to voice is not a migration. It is a rebuild.