Blog Details

"The voice agent feels slow" is not an actionable bug report. Voice agent latency is the sum of several independent stages, each with different bottlenecks, and fixing the wrong one wastes engineering time without moving the number users actually feel. This is a methodology for measuring each stage in isolation so you know exactly where the time goes before you optimize anything.

The Pipeline You're Measuring

A typical voice agent turn looks like this:

[User speaks] → [STT] → [LLM] → [Tool/API calls] → [TTS] → [User hears response]
     │             │        │           │              │
   VAD end      transcript  first     external      first audio
   detected     available   token     data fetch     byte out

The metric users feel is end-of-speech to first-audio-out — not total request time, not time-to-full-response. Optimize for the perceived-latency metric, not an easier-to-measure proxy.

Instrumentation: Timestamp Every Boundary

Add timestamps at each transition, not just start/end of the whole turn:

import time

timings = {}

def mark(event):
    timings[event] = time.monotonic()

# In the pipeline:
mark("vad_end_of_speech")
transcript = stt.transcribe(audio)
mark("stt_complete")

response_stream = llm.generate(transcript)
first_token = next(response_stream)
mark("llm_first_token")

full_response = "".join([first_token] + list(response_stream))
mark("llm_complete")

audio_chunk = tts.synthesize_first_chunk(full_response)
mark("tts_first_audio")

From these timestamps, compute the metrics that actually matter:

• STT latency: stt_complete - vad_end_of_speech
• LLM time-to-first-token (TTFT): llm_first_token - stt_complete
• LLM full generation time: llm_complete - llm_first_token
• TTS time-to-first-audio: tts_first_audio - llm_complete (or - llm_first_token if streaming into TTS)
• Total turn latency: tts_first_audio - vad_end_of_speech

Why Time-to-First-Token and First-Audio Matter More Than Totals

If your architecture streams LLM tokens into TTS incrementally (rather than waiting for the full LLM response before starting synthesis), the perceived latency is dominated by LLM TTFT + TTS TTFA, not total generation time. Two architectures with identical total latency can feel completely different to a user:

Architecture LLM TTFT LLM full gen TTS starts after Perceived latency Non-streaming 300ms 1200ms full LLM response ~1500ms+ Streaming (token→TTS) 300ms 1200ms first LLM token ~300-500ms

Same total compute time, dramatically different user experience. If you're not streaming tokens into TTS, that's usually the single highest-leverage fix before touching model choice or infrastructure.

Isolating External API Call Latency

Tool calls (checking a calendar, looking up an order, querying a database) sit inside the LLM's generation loop and often dominate turn latency without showing up clearly unless you instrument them separately:

def timed_tool_call(name, fn, *args):
    start = time.monotonic()
    result = fn(*args)
    duration = time.monotonic() - start
    log_metric(f"tool_call.{name}.duration_ms", duration * 1000)
    return result

Track P50/P95/P99 per tool, not just an average — a tool call that's fast 95% of the time but occasionally takes 3 seconds (cold database connection, upstream rate limiting) is often the actual cause of "sometimes it's slow" complaints that don't reproduce in casual testing.

A Benchmark Harness

To compare STT/LLM/TTS provider or model choices objectively, build a fixed benchmark set:

1. Record 20-30 representative user utterances (varying length, accents, background noise if relevant to your product).
2. Run the same utterances through each candidate configuration.
3. Report P50 and P95 for each stage, not just averages — tail latency is what users notice as "occasionally really slow."

Benchmark results (P50 / P95, ms):
                  STT        LLM TTFT     TTS TTFA     Total
Config A (baseline)   180/240    310/520     140/210    630/970
Config B (smaller LLM) 180/240   140/260     140/210    460/710
Config C (local STT)   90/130    310/520     140/210    540/860

This table format makes tradeoffs explicit — Config B trades LLM quality for latency, Config C trades STT infra complexity (self-hosted) for latency. Decide with the actual numbers in front of you, not intuition.

Common Latency Traps

• Cold starts: serverless STT/LLM/TTS endpoints often have materially higher P95 latency due to cold starts. Measure with realistic idle-then-request patterns, not back-to-back warm requests.
• Network round-trip to a distant region: if your inference endpoints are in a different region than your application server, that's fixed latency added to every stage. Co-locate where possible.
• Blocking on the full STT transcript before starting LLM generation: streaming STT partial transcripts into the LLM (with a final-transcript confirmation) can shave meaningful time off TTFT for longer utterances.
• Synchronous tool calls in the critical path: if a tool call isn't needed to start the spoken response (e.g., you can say "let me check that" while fetching), don't block TTS behind it.

Reporting Latency Internally

Report these four numbers as your standard voice agent latency dashboard, tracked over time and per release:

• STT latency (P50/P95)
• LLM TTFT (P50/P95)
• TTS TTFA (P50/P95)
• End-to-end turn latency (P50/P95)

This turns "the agent feels slow" into a specific, attributable regression the next time it happens, instead of a vague investigation from scratch.