How long should a customer wait for an AI phone agent?

Every voice AI vendor quotes a response time. Almost none of them tell you what that number is measured on, which is a problem, because on a real deployment the dominant cost is usually nothing to do with the model.

We measured ours properly. Here is what came out.

The numbers

turn 1   4993 ms   routes the caller, then checks stock
turn 2   2309 ms   same specialist
turn 3   2051 ms   same specialist

Two seconds is fine. Five is not. A caller hears five seconds of silence and starts saying "hello?", which then collides with the agent's reply and the call goes sideways.

The important detail is that the first turn is the expensive one, and the first turn is exactly where a caller decides whether this is worth their time.

Where the time actually goes

Breaking that number down changed what we worked on:

Network distance to the inference endpoint: around 1.5 seconds per model round trip. This is geography, not code.
Tool execution: 4 to 7 milliseconds. Looking up stock, reading the diary, writing to the CRM. Effectively free.

Read those two lines together. The work our software does accounts for well under one percent of what the caller waits for. Any optimisation inside the application is noise. Only changes that remove a round trip, or overlap one with something else, move the number at all.

Two things that did not help

While the measurement harness was out, we tried the obvious fixes.

Switching models. We tested several. The spread between them was far smaller than the spread between architectures. Picking a different model is the first thing everyone reaches for and it bought us the least.

Shortening the prompt. A three-word prompt takes the same time as a 1,200-token one. We ran it twice because the result was so counterintuitive. Prompt length is not where the time goes.

What did help

Removing a decision. Our first architecture routed each call to a specialist agent. That routing step is a whole extra model round trip, paid on the first turn. Removing it took the first turn from 4993 ms to 3217 ms, with no loss in accuracy on our evaluation suite.

Overlapping the wait. Generating a response and starting speech before the full turn resolves hides a round trip behind speech that was happening anyway.

Tuning when the agent decides you have stopped talking. A lower endpointing delay takes real time off every single turn.

What to ask a vendor

If someone quotes you a response time, three questions separate a real number from a demo number:

Is that the first turn or a later one? They are very different, and the first is the one that loses callers.
Measured from where? A number from a data centre next to the model provider will not survive contact with your actual deployment.
What is in it? Model time, tool time and network time behave completely differently, and only one of them is usually theirs to fix.

A vendor who has measured their own system can answer all three without hesitating. One who cannot has probably never looked.