Work cited in Deepgram's guide puts tool calls at 30 to 80% of the latency before a voice agent starts its final answer. If you only log the total turn time, you can't see that, and you can't tell which stage is making the agent feel laggy. The questions below cover what to measure, what to aim for, and which lever to pull first.
What counts as latency in one turn, and what is a realistic target?
The clock starts when the caller stops speaking and ends when they hear the bot. That is the turn gap, and it's the number the caller actually experiences. Deepgram's guide suggests keeping the 95th percentile turn under about 1,000 ms, as summarized in this guide to voice bots on real phone lines.
Deepgram's guide also gives planning figures for each stage, but they use different measurement boundaries. One stage may be timed from the last audio frame and another from the final transcript. Treat the figures as a starting yardstick, not numbers you can add up to 1,000. The only budget that holds is the one built from your own logs, with boundaries you define.
Why do the 200 ms and 700 ms figures matter for a bot?
The guide cites research across ten languages in which response timing to questions peaks between 0 and 200 ms. That sample didn't include an Indian language, so read it as a broad human baseline. It's a reference point, not a target to chase.
The 700 ms figure matters more in practice. Corpus work the guide cites links gaps of 700 ms or more with disagreeing or declining responses. A bot that always pauses about a second can sound reluctant even when the answer is yes. That reading is an inference, but it follows: a late "yes" may land as a hesitant one, so the slowest turns do more damage than the average suggests.
Which stages and levers move the budget most?
A turn passes through end-of-turn detection, speech to text, the model's first token, any tool calls, and the start of text-to-speech audio. Network placement sits underneath all of them. The guide says transport, transcription, and the model's first token each take a large share. Tool calls can take the biggest one, at 30 to 80% of the latency before the final answer starts, per the work Deepgram's guide cites.
That range is wide, and that's the point. If your agent calls tools, check them first. If it doesn't, look at end-of-turn detection, because a fixed silence timer adds its full length to every turn.
| Lever | What to do | Trade-off | |---|---|---| | End-of-turn detection | Replace a fixed silence timer with a model that weighs meaning and sound | Cutting the timer alone makes the bot interrupt people mid-thought, which feels worse than waiting | | Model first token | Pick by time to first token, then stream | A faster model may reason less well, so move hard reasoning into a backend call | | Tool calls | Speak a short bridge such as "let me check that" and set a timeout on every tool | The bridge hides the wait but doesn't shorten it, so keep fixing the slow tool | | Text-to-speech start | Stream audio from the first words of the reply | Check how numbers and amounts are read, since $150,000 read digit by digit loses the caller quickly | | Placement | Keep media, speech, and reasoning services close together and near callers | Regional capacity for a given model may be limited, so check it early |
What should the per-turn log look like?
Log each boundary as a timestamp from one clock, then compute the stages by subtraction. Mixing clocks across services can produce negative durations. The table lists the boundaries to record. We cover related ground in see also: sharp vs cloudinary vs imgix for fixed avatar sizes.
| Stage | Start boundary | End boundary | Record | |---|---|---|---| | End-of-turn detection | Last frame of caller speech | Turn declared finished | Duration, detector type, whether the caller resumed | | Speech to text | Turn declared finished | Final transcript ready | Duration, transcript length, confidence if exposed | | Model first token | Request sent | First token received | Duration, model name, prompt size | | Tool call | Tool request sent | Tool result returned | Duration per tool, timeout hit or not | | Text-to-speech start | First text sent to speech | First audio byte received | Duration, voice, text length | | Send to phone leg | First audio byte received | First audio frame written to the call | Duration, transport (SIP or WebSocket) |
The last row ends at your server, not at the caller's ear. Carrier and handset delay sit beyond it, so the logged number slightly understates what the caller hears. That's an inference from where the boundary sits, and it's a reason to compare against a recorded test call now and then.
Here is a minimal TypeScript sketch of the idea. It is unexecuted, so run it against your own pipeline before trusting it. It has no dependencies beyond a runtime that provides performance.now(), such as Node.js or a browser.
const ORDER = [
"caller_last_speech",
"turn_declared",
"transcript_final",
"llm_first_token",
"tts_first_byte",
"audio_sent",
] as const;
type Mark = (typeof ORDER)[number];
class TurnTrace {
private marks = new Map<Mark, number>();
toolMs = 0; // sum of tool call durations inside this turn For more on this, see [a closer look at rds inaccessible encryption credentials: kms triage guide](/rds-inaccessible-encryption-credentials-kms-triage-guide).
mark(name: Mark, at = performance.now()) {
if (!this.marks.has(name)) this.marks.set(name, at);
}
report() {
const stages: Record<string, number | null> = {};
for (let i = 1; i < ORDER.length; i++) {
const a = this.marks.get(ORDER[i - 1]);
const b = this.marks.get(ORDER[i]);
stages[ORDER[i]] = a !== undefined && b !== undefined ? b - a : null;
}
const first = this.marks.get(ORDER[0]);
const last = this.marks.get("audio_sent");
const gap = first !== undefined && last !== undefined ? last - first : null;
return { gap, toolMs: this.toolMs, stages };
}
}
Expect one object per turn with a gap field, a toolMs field, and one entry per stage. To verify, check that the stage values add up to gap when every mark is present. If they don't, a mark is firing out of order.
Then test the failure cases on purpose. A turn with no transcript, or an early barge-in, should produce null for the missing stage, not a fake zero or a negative number. If your pipeline streams, a stage such as the first token can start before the final transcript. Pick a boundary and keep it fixed.
Also read: see pagespeed insights score changed without site changes?
How should you set targets and review the slow turns?
Start with the vendor yardstick for p95, about 1,000 ms, and treat it as a first guess. The guide gives no p50 target, so set yours from a small pilot of real calls. Aim for a median well under that p95, and watch the spread between the two. A wide spread means a few turns are doing the damage.
Review the slowest turns every week, not the average. A caller remembers the four-second silence, as the guide puts it. Sort turns by gap, open the top ones, and read the per-stage log. If one stage dominates the slow turns but not the typical ones, you have a tail problem, often a tool or a cold connection. If every stage is a little slow, look at placement.
For tools, set a timeout on each one and speak a bridge before the call starts. One illustrative setup: queue a bridge phrase as the tool request leaves, and pick the timeout from that tool's own measured p95. Log whether the timeout fired. The bridge protects the caller's patience but doesn't shorten anything, and a bot that bridges every turn has only moved the problem.
Is a faster model the fix for a slow voice agent?
Usually not on its own. The common assumption is that the language model owns the delay. The model's first token is one large share among several, and the guide says the same of transport and transcription. A turn that waits on a slow tool or a long silence timer stays slow with any model.
A second assumption is that stage figures from a vendor guide add up to your total. They don't, because they use different boundaries. Log your own stages from one clock, compare p50 with p95, and change one lever at a time. That's the only way to tell a slow model from a slow network.



