AI voice agents for restaurants: why the first turn feels slow (and how I fixed it)

The problem, as reported by humans

I run a voice ordering agent for a restaurant — a real-time conversational voice AI that answers the restaurant's phone, takes pickup and delivery orders, confirms them with the caller, and hands the order to the kitchen. The pipeline is standard for 2026: SIP call comes in, streaming STT transcribes the caller, an LLM decides what to say and which tools to call, streaming TTS speaks the reply. All of it lives in a real-time voice agent framework that stitches the vendors together.

The bot was in acceptance testing with a client. He kept saying the same thing:

"It feels sluggish. Like it hesitates after I say 'Pickup.' Then the rest of the call is fine."

This is a bad bug report. Not because the client is wrong — he isn't; the pause is real and I could hear it too — but because "sluggish" and "hesitates" cover about five different subsystems in a voice AI pipeline, each with different remediation cost. Before I write any code, I need to know which subsystem is actually slow.

This article is about the diagnosis, not the fix. The fix turns out to be trivial once you know what's broken. Getting to what's broken is the interesting part, and it's the transferable skill for anyone building AI voice agents for restaurants, hotels, clinics, call centers, or any other domain where a real-time voice bot answers a live phone call.

Why "the voice AI is slow" is a useless framing

A turn in a voice agent isn't one event. It's a chain of at least seven observable segments, each of which can independently add latency in a real-time voice AI pipeline:

Caller stops speaking — the acoustic event at the caller's phone.
Endpointing decides the turn is over — some combination of VAD (voice activity detection) silence and, in modern stacks, a semantic end-of-turn model on the STT side. This is a decision, not an observation. It always waits a bit.
STT emits a final transcript — up to this point most STT systems have been emitting interim results. The moment the transcript freezes and gets marked "final" is when downstream code starts working.
First token from the LLM (TTFT) — the model receives the prompt, does its internal work, and emits the first output token.
Full LLM response — all tokens generated, including any tool calls.
First byte from the TTS (TTFB) — the synthesized audio starts flowing.
First audio packet reaches the caller — network hop back through the SIP path.

If someone tells you "it's slow" and points at any of these, they're guessing. The whole point of building a real-time pipeline is that you have telemetry for every one of these segments, per turn. Use it.

Here's what I want in a bug report before I touch any code:

Segment	How I measure it	What "slow" means here
Endpointing	`caller_last_speech_ms → stt_final_emitted_ms`	Configured delay is too generous, VAD too twitchy, semantic-EOT model too cautious
STT final	`stt_final_emitted_ms → llm_request_sent_ms`	Something between STT and LLM (tool schema build, prompt assembly) is blocking
LLM TTFT	`llm_request_sent_ms → llm_first_token_ms`	Vendor side: cold cache, rate limit, quota throttling; or a huge prompt
LLM full	`llm_first_token_ms → llm_last_token_ms`	Token generation speed, or LLM produced a long reply
TTS TTFB	`tts_request_sent_ms → tts_first_audio_byte_ms`	Vendor cold start, voice loading, network to TTS
Network to caller	`tts_first_audio_byte_ms → caller_hears_ms`	RTP/SIP path, usually small and stable

If your voice AI agent doesn't emit these six numbers per turn, stop reading this article and go add them. Everything downstream depends on them. This is the single highest-leverage instrumentation change you can make in any real-time voice bot, whether it takes restaurant orders, hotel bookings, or medical intake.

What the trace actually showed on the restaurant voice bot

Once I had per-segment numbers, the pattern was immediate.

For the client's complaint — "hesitates after I say Pickup" — the turn breakdown looked like this on a fresh call:

Segment	Time
Endpointing (after "Pickup.")	~1.0 s
STT final emit	negligible (streaming STT keeps overhead small)
LLM TTFT	~2.2 s
LLM full	~0.6 s (short reply)
TTS TTFB	~0.13 s
Network back to caller	small
Total, caller "Pickup" to bot speech	~4.9 s

Endpointing takes about a second by design — I intentionally give callers time to continue if they meant to say more. STT is streaming, so the "final" event is nearly instant once endpointing decides the turn is over. TTS is a well-behaved commercial vendor with tight first-byte latency. The elephant is the LLM: 2.2 seconds to first token.

That was suspicious in itself. On the same call, later turns showed a different picture:

Turn	LLM TTFT
Turn 1 (right after "Pickup")	2.2 s
Turn 2 (right after "cheese pizza and garlic knots")	0.9 s
Turn 3 (right after "that's all")	0.7 s
Turn 4 (right after "yes")	0.6 s
Turn 5 (name)	0.7 s

Same model, same prompt, same tool schema, same network path. Only the first inference is slow. If it were a fundamental prompt-size problem, every turn would be slow. If it were rate limiting, it would be random. If it were network, it would jitter.

The pattern says: the first request on a fresh call is doing something the later ones aren't.

What was different about the first request — prompt caching for voice AI

At this point the answer is one Google search away, but let me walk through the reasoning because that's what transfers.

Modern commercial chat LLM APIs — OpenAI, Anthropic, Google, and the open-model providers that mimic their APIs — cache the prefix of the input. The prefix in a voice AI agent is: system prompt + tool schema + any static context. In my case, the system prompt for the restaurant voice bot is about 13 000 tokens. It contains the full menu, dozens of behavior rules for common failure modes (how to handle "add another one," what to do when the caller asks for something not on the menu, how to read back a phone number for SMS confirmation), and the JSON schema for the twelve tools the LLM can call.

Caching keys off the exact byte content of that prefix plus the API key and model. First request in a fresh cache window → full 13K tokens have to be tokenized, embedded, and processed. Every subsequent request against the same prefix → the vendor serves the prefix from its internal KV cache. That's what the metric cached_tokens in the completion response tracks, and it's the single most useful field for diagnosing this.

I looked at the actual response bodies:

Turn	prompt_tokens	cached_tokens	TTFT
Turn 1 (first LLM call on the call)	13 009	0	2.22 s
Turn 2	13 088	11 008	0.93 s
Turn 3	13 247	13 056	0.71 s

Cache miss on the first inference. Cache hit — 85%+ of the prefix — from turn two onward. The pause the client heard is the cost of that miss.

There are a few reasons this happens specifically on the first turn of a call and not before:

The greeting the caller hears first is spoken by TTS with no LLM in the loop. In my pipeline the greeting is a deterministic string — the framework's session.say() — sent straight to the TTS vendor. No LLM call happens until the caller replies. So the first LLM inference of the whole call comes when the caller finishes their first sentence, not when the call connects.
Cache windows on commercial LLM APIs are short, measured in minutes, not hours. On a low-traffic bot (or when a fresh worker process picks up the call), the cache is cold by default.
Workers restart on every deploy, and so does the effective cache warmness at the vendor for that combination of prompt + tools + key. A deploy 30 seconds before the call has the same effect as no traffic for an hour.

All three converge on the same wall-clock moment: the client hears "Pickup" and waits for the bot's first reply.

The fix: prompt-cache warmup for real-time voice agents

Now the fix is boring, because we know exactly what's happening. We need the prompt prefix to be warm at the LLM vendor by the time the first real inference fires. This is generic to any voice AI that hits a chat LLM API — restaurants, hotels, medical intake, customer support voice bots — the pattern below transfers verbatim.

There are three tactics:

Warm at process start. When a voice agent worker boots up, send one throwaway inference against every restaurant's system prompt with a small completion cap. The vendor caches the prefix. The next real call from that worker gets a hit.
Warm at call start. When the call connects and the greeting starts playing, fire a warmup request in parallel with the greeting audio. The caller listens to "Hi, you've reached ..." for two seconds; that's the window we use to make the vendor cache warm. The caller says "Pickup." at second three; by then the cache is hot.
Do both. Belt and suspenders. Process-level warmup covers workers with recent traffic; per-call warmup covers everything else, including the awkward "just deployed 30 seconds ago" case that hits every AI voice agent for restaurants after any code push.

The warmup request itself is a one-line call to the chat completion API:

warmup = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": system_prompt},   # BYTE-IDENTICAL to real call
        {"role": "user", "content": "warm"},
    ],
    tools=tools_schema,                                  # BYTE-IDENTICAL
    max_completion_tokens=100,                           # reasoning-model floor; adjust for your model
)

Three things I got wrong on the first attempt, worth calling out:

The system prompt has to be byte-identical to what the real call will send. One extra whitespace, one dynamic timestamp, one different field in the tool schema, and you get a cache miss and paid for nothing. Use the same function that builds the real prompt.
The tool schema is part of the cache key. If your voice framework wraps tool definitions differently from what you're sending in the warmup, you cache the wrong prefix. Match whatever your live path sends, right down to the wrapper.
Some model families require enough token budget to finish "internal thinking" before emitting any output token. If you set max_completion_tokens=1 on a reasoning-family model, the vendor returns a 400 saying "output truncated before any token was emitted." A hundred is fine and costs almost nothing on prompt-input pricing.

Everything else is engineering: put both warmups behind a feature flag, swallow all exceptions inside the warmup path (a broken warmup must never break a real call), and log the outcome so you can watch cache hit rates rise.

The result

Same client, same test scenario, same phone. Warmup on:

Metric	Before	After	Change
First LLM TTFT	2.22 s	0.80 s	−1.4 s
`cached_tokens` on turn 1	0	12 500+	90%+ hit
Caller "Pickup" → bot speaks	4.9 s	2.6 s	−2.3 s

That's the whole intervention. The remaining 2.6 seconds are structural — endpointing waits ~1 s deliberately, then there are two LLM inferences on that turn (one for the tool call, one for the spoken reply, because I have parallel_tool_calls=False for deterministic behavior), plus TTS. Those are separate optimization targets with their own cost/benefit conversations. But the specific, client-audible "hesitation" is gone.

Cost of the warmup, calculated conservatively: about $0.02 per call on the input-token side of my vendor's pricing. On a bot that handles two-minute calls with 40 inferences each, that's a couple of percent on the LLM cost line. Nothing.

What to do with this if you're building an AI voice agent for restaurants (or anything else)

Some of this generalizes; some of it is specific to my stack. Here's the transferable part.

Instrument the seven segments before you do anything else. If your voice AI framework doesn't give you per-turn per-segment timing, add it. This is 30 minutes of work and it's the difference between diagnosis and guessing. Every real-time voice bot benefits — restaurant ordering, hotel reservations, medical triage, insurance quoting — they all share the same STT-LLM-TTS anatomy and the same failure modes.

Look at first-turn behavior separately from steady-state. The most common first-turn problems are: cold LLM cache (this article), cold TTS voice model (rare on modern vendors), worker warm-up (framework-dependent), and DNS/TLS handshake to vendor endpoints (usually solved by a preflight in the worker's boot sequence). All four have different fixes. Don't lump them.

When the LLM segment is the outlier, look at cached_tokens first. Every major commercial chat LLM API in 2026 returns this field. If it's zero on requests that should hit an existing prefix, you have a cache identity problem — usually a byte drift in the prompt or the tool schema. If it's high but TTFT is still slow, it's not caching; it's something else (throughput, region, prompt actually is enormous). The number tells you which conversation to have.

Warmup is cheap. Redesigning your prompt is not. People jump to "shrink the system prompt" when they see cold TTFT. That's usually a bigger scope than they think — every rule in the prompt was written to fix a bug and removing it costs a QA cycle. Warmup addresses the symptom for pennies and lets the prompt stay well-tested. If TTFT is still bad after warmup, then talk about the prompt.

Match your warmup shape to your live shape exactly. I got this wrong twice — once on the tool wrapper shape, once on the token budget for a reasoning-family model. Both times the warmup call ran and swallowed its own error, so the metric said "warmup succeeded" while the actual cache stayed cold. Assert on cache hit rate in production, not on warmup success rate.

Don't touch parallel_tool_calls casually. People pattern-match "two inferences on the first turn" to "just turn on parallel tool calls." In a voice agent where your tool results contain instructions the model is supposed to obey verbatim — "say exactly X" — parallel tool calls give the model permission to speak before it sees the tool result. That path breaks deterministic copy in ways you won't see for weeks. If you have parallel_tool_calls=False for a reason, keep it and optimize elsewhere.

What this doesn't fix

The remaining 2.6-second window has three structural pieces. In case you're facing the same tradeoffs:

Endpointing delay exists because you don't want to cut callers off mid-sentence. Modern semantic-EOT models let you go under 500 ms in some cases, but the tradeoff for aggressive endpointing is that compound orders get chopped. I keep mine at about a second and consider that a UX budget, not a bug.
Two sequential LLM inferences per turn with a tool call is structural to the design of most tool-using chat models. You can eat it, or you can move to a deterministic shortcut for the most common first replies. In my case, "Pickup" and "Delivery" cover essentially 100% of first turns in production. A dedicated intent-classifier-plus-precomputed-audio shortcut would take another second off, but at the cost of new code paths that need their own QA. Whether that trade is worth it is a product decision, not a technical one.
TTS first byte is already at ~130 ms with my vendor and I don't see room. Different vendors are different; benchmark yours.

I mention these because if you follow the advice above and get first-turn TTFT under a second, and callers still complain the bot feels slow, these are what's left.

The one-line takeaway for voice AI latency

If your AI voice agent's first turn feels slow and later turns don't, look at the LLM's cached_tokens field on the first inference. It's almost certainly zero. A warmup request that shares byte-for-byte the same prefix as your live call, fired parallel to the greeting audio, closes the gap for pennies. Everything else is engineering hygiene.

The reason the client heard "hesitation" on the restaurant voice bot wasn't STT, wasn't TTS, wasn't my code. It was a vendor-side prompt cache being cold on the one request per call where a human is listening for the response. Once I could name it, the fix was thirty lines. Naming it required breaking the turn into segments and looking at each one, which is the actual skill and the actual point of this write-up.

If you're building a real-time voice pipeline — for restaurants, hotels, healthcare, retail, or any domain where an AI voice agent takes live phone calls — the segment breakdown is worth adding today even if you don't have a complaint yet. When the complaint comes, you'll be looking at numbers instead of guessing.