Voice AI - The Next Frontier | Decagon, Retell, Vapi, Smallest AI, Daily

About this episode

Voice agents are one of the hottest use cases in enterprise right now, but also one of the hardest to actually take live. Getting latency low enough to feel human without dumbing down the responses, making reliable tool calls to a CRM without dropping the customer mid-call, building fallback models for when Anthropic or OpenAI are running hot. None of it is as simple as the demos make it look.

Last week I hosted a fireside chat with five eng leaders who deal with this stuff every day: Basia Sudol (Head of Enterprise Solutions, Decagon), Varun Singh (CPTO, Daily), Steven Diaz (FDE Manager, Vapi), Tyler D'Silva (Founding FDE, Retell AI), and Sudarshan Kamath (Founder, Smallest AI).

We get into why nobody serious is shipping real-time voice-to-voice yet, why LLMs forget the middle of your prompt (and what that does to your architecture), why a giant prompt quietly destroys your unit economics, and why voice agent costs are now being compared directly against human labor.

Plus the stuff nobody warns you about: turn-taking, HIPAA constraints, why outbound is easier than inbound, why getting an exec to actually like the voice can be harder than any model problem, and more!

Chapters below:

00:00 Intro

00:26 Meet the panel

01:23 Daily, WebRTC, and 20 years of building voice

03:49 How Smallest AI made real-time TTS work

05:23 Why Decagon moved into voice

08:16 How Vapi and Retell think about the stack

11:14 Forward deployed vs solutions engineering

16:48 Voice agent architecture, explained simply

22:11 Cascade vs speech-to-speech: the real tradeoff

28:11 Hybrid pipelines and mixing models

32:16 Accents, multilingual, and getting Singlish right

35:32 Prompts vs workflows, and the latency fight

44:26 How you actually evaluate a voice agent

49:04 Simulation-based evals

49:49 What production metrics really look like

51:53 Building a QA framework that scales

54:53 Evaluating speech-to-speech

58:10 Open source benchmarks

59:57 Why picking a voice is so subjective

01:02:03 Personalization and custom voices

01:03:20 Voice quality is solved, GPU efficiency is the new war

01:06:23 Why outbound calls work better than you'd think

01:10:38 Deploying in regulated industries (HIPAA, retention, audits)

01:12:43 Turn-taking, the hardest unsolved problem in voice

01:18:53 Where voice agents go in the next year

01:27:55 Audience Q&A: inside Smallest's Hydra model

01:32:15 The deployment problems nobody has solved yet

01:37:46 Closing thoughts and thanks