Case study: a phone agent that books the appointment and files the paperwork

Two things go wrong with business phone calls. The ones that come in after hours are never answered, and the ones that are answered are never written down. The second is the expensive one. A call that nobody logs is a customer nobody follows up.

This is a walk through a voice agent built to fix both, and through the one decision on the project that changed the product more than any model choice did.

What it had to do

The brief was not "answer the phone". Answering is easy and there are a dozen products that do it. The brief was that a call should leave the business in a different state than it found it.

So the agent had to:

• Work out who is calling and pull up their record
• Answer questions about stock and orders from live ERP data, not a static FAQ
• Book an appointment into a real calendar
• Write the contact, the deal, the follow up task and a call summary into the CRM

That last line is the whole point. Everything before it is conversation.

What it does now

A caller rings the number. The agent identifies them against HubSpot, and if the record is wrong it corrects it rather than creating a duplicate. It answers stock and order questions by calling the ERP directly. Policy questions come from retrieval over the company's own documents rather than from the model's memory, so the answer is the company's answer.

When the call ends, HubSpot has the contact, the deal if one opened, a follow up task if one is needed, and a summary of what was actually said.

The ERP sits behind a service interface rather than being called directly. That sounds like housekeeping, and it is, but it means the backing system can be replaced without rewriting a single one of the agent's tools. Businesses change ERPs. Agents should survive that.

The decision that mattered

The standard advice for an agent with many skills is to split it into specialists behind a router: a sales agent, a stock agent, a booking agent, and something at the front deciding where each caller goes. The reasoning is sound. A shorter menu of tools is chosen from more reliably.

That version was built first. Then it was measured against a single agent holding every tool at once.

	specialists	one agent
eval score	10/10	10/10
first turn	4993 ms	3217 ms
later turns	2180 ms	2853 ms
first turn penalty	2.3x	1.1x

Identical accuracy. The routing step is a whole extra model round trip, paid at the start of every call, buying nothing.

Worse, the split introduced two failures that a single agent cannot have. A capability reachable from only one specialist stranded callers: someone ringing to correct the spelling of their name got passed between agents instead of helped. And two specialists could hand a request neither could serve back and forth until the caller gave up.

So the architecture was deleted. The one that shipped is the simpler one, and it is faster where callers actually notice: the first answer, when they are deciding whether this is going to be worth their time.

The split is still in the codebase, still tested, and worth re-measuring if the tool count ever grows well past a dozen. Deleting it was a decision about today's numbers, not a permanent opinion.

Where the time actually goes

Before optimising anything, it was worth finding out what was slow.

tool execution        4 to 7 ms
model round trip      about 1500 ms

The code was never the bottleneck. Network distance to the inference endpoint was, and no amount of tuning application logic touches that.

Two popular fixes were measured and did not help. Switching models produced no useful gain across the ones tested. Shortening prompts did nothing at all: a three word prompt takes the same time as a twelve hundred token one.

What does help is overlapping the wait with speech that has to happen anyway, and tuning how long the agent waits before deciding the caller has finished talking.

The general point is duller than the specific one. If nobody has measured, the optimisation is guesswork, and guesswork usually lands on the part of the system that is easiest to see rather than the part that is slow.

How it is kept honest

Twelve eval cases run from the command line. Each one names the tool the agent must call, the tools it must not, and what the answer has to contain. They either pass or they fail loudly.

That harness is unglamorous and it is the only reason the architecture comparison above was possible at all. Without it, "the specialists feel better" would have been an opinion, and opinions do not lose arguments to data they cannot see.

There is also a dashboard showing every call: what was said, which tools were invoked, how long each took, and what failed. When a customer says the agent got something wrong, the answer is a record rather than a theory.

What this means for you

If you are looking at a voice agent for your business, the questions worth asking are not about the model.

• Does it write back? An agent that answers well and records nothing is a more expensive answering machine.
• Has anyone measured it on your deployment? Vendor latency numbers are measured somewhere with a good connection to a data centre. Yours may not be.
• Can you see what it did? Months later, on a specific call.
• What happens when the backing system changes? If swapping your ERP means rebuilding the agent, that is a cost nobody quoted you.

None of that requires a better model. It requires somebody to have measured, and to have been willing to throw away the version they had already built.