At localhost, Griffin Sharp, applied AI engineer at AssemblyAI, demoed a way to automatically surface and diagnose voice agent issues at scale. AssemblyAI builds the models, APIs, and infrastructure behind voice-powered applications.
Griffin works with customers running voice agents in production: ordering at a drive-thru, taking new patient intake calls for a doctor's office, triaging support issues live over the phone. They range from bootstrapped startups to Fortune 500 companies, each with a different audio stack, each running anywhere from thousands to hundreds of thousands of real calls a day.
The feedback loop is broken
A team ships a change, does some quick manual testing on staging, and it looks fine. In production, calls run into noisy audio, accents, crosstalk, and short utterances that staging never surfaced. There's rarely a clean way to catch that early. Instead, the signal comes from customers, and only the ones engaged enough to notice something's wrong and say so.
Even with good logs, transcripts, and recorded audio in hand, the underlying issues resist a simple diagnosis. They aren't errors with codes and clear causes. A conversation can sound completely plausible end to end and still fail the person on the other end of it: a clipped utterance, a mishandled interruption, a dropped entity, or a handful of small errors compounding across speech recognition, orchestration, generation, and text to speech. Manually reviewing enough calls to catch a pattern like that doesn't scale.
The goal is a pipeline that catches configuration problems before users do: something fast, automated, and intelligent enough to run on logging a team probably already has, plus a small set of AssemblyAI tools layered on top, so that the time freed up goes toward the failures that actually need a person: mistranscription, hallucination, bugs, and the engineering work only a human can do.
Fifty calls, re-examined
The demo started from 50 recorded calls from a patient-booking voice agent, collected over roughly an hour through AssemblyAI's real-time transcription. Each call already had logging in place: connection events, config parameters, partial and final transcripts as they arrived, voice activity detection (when a speaker actually started talking versus when a transcript showed up for it), speaker identity, and other orchestrator events. At the end of that window, each call carried a real-time transcript, a full log, and a complete audio recording, ready for a fleet-level analysis to kick off, on a schedule or, for the demo, by hand.
Live transcription is built to serve a caller who's waiting; every decision in that path is optimized for latency. Quality analysis has a different job entirely: revisit the audio after the fact, spend more time and more context on it, and check what was actually said against logs and deployment history. That distinction is why the pipeline re-processes each call rather than trusting the real-time transcript alone
The first step re-transcribes the full recording with AssemblyAI's asynchronous API, using the Universal 3 Pro model. Freed from the real-time latency constraint, this pass gets the complete recording with speakers separated per channel, producing a more reliable transcript than the live one. Any disagreement between the two transcripts is itself a signal worth digging into. The second step sends that transcript to AssemblyAI's LLM Gateway for a quick summary: what information the call collected, whether the booking succeeded, what the outcome was. The third step sends a second LLM Gateway request, this time a rubric-based diagnostic that pulls in everything available for the call, the live transcript and logs, the re-transcribed ground truth, and the summary, and scores it against a defined set of failure patterns: interruptions, split sentences, dropped digits, the agent mishearing something, slow turn-taking. The fourth and final step is a fleet synthesis: one request that aggregates every per-call diagnosis alongside recent deployment history and asks the pipeline's real question: what, if anything, went wrong across the fleet.
Fifty calls became 201 runs
Each of the 50 calls runs through this pipeline as its own chained Render Workflow, all 50 kicking off at once. Every call gets one parent task, Process Call, with three child tasks chained in order: transcribe, summarize, score.
"For each task run, Render is going to spin up a fresh container, execute it, and tear it down." Each step gets its own compute on demand and its own retry policy, so a failed summarization step retries just that step instead of re-running (and re-paying for) the transcription before it. Once all 50 calls are scored, a separate final task, Fleet Synthesis, queries Postgres for every call's diagnostic and writes the aggregate report.
Add it up: 50 parent task runs, 150 child task runs, and one synthesis task, 201 task runs total, from a single Render Workflow service. The application decides how much compute this needs at the moment, not a fixed worker pool sized in advance: the workload shows up in bursts, and its size scales with however many calls came in. AssemblyAI supplies the speech and language models doing the actual analysis; Postgres keeps every call's evidence around for whoever needs to look at it later.
A diagnosis, not a verdict
In this run, the synthesis step landed on a specific answer: a recent deployment likely caused a major endpointing regression. Phone number and other entity-capture failures rose from 7 percent to 83 percent in the 50-call set, correlated directly with the deployment. The recommendation wasn't a full rollback: a specific setting, the minimum turn silence AssemblyAI uses to decide when a speaker has finished talking, had been set too low, cutting people off mid-sentence, and the fix was simply to raise it.
The key word here is "likely." A rubric like this can rank suspicious patterns and attach evidence to each one, but it can't prove causality on its own. Someone still has to reproduce the failure, check the actual configuration change, and confirm the fix holds across different accents, noise conditions, and call types. What the pipeline actually delivers is much narrower and far more useful than a stack of 50 individual transcripts to comb through by hand: one testable hypothesis, a concrete failure rate, and the exact set of calls needed to check whether the fix works.