Prompt Caching & Latency Optimization

Overview

Two related but separate concerns affect how an assistant feels and costs to run: how much you pay per minute of LLM usage, and how quickly the assistant responds. This page covers prompt caching (which affects cost) and several techniques for reducing perceived latency, including flush syntax and immediate acknowledgment messages.

Cached input refers to stable prompt content that a model provider can reuse across requests, typically at a discounted rate compared to processing it fresh each time. Cost estimates blend the standard and cached token rates into a single effective input rate, and that blend assumes a 50% cache-hit rate.

Because the 50% figure is only an assumption baked into the estimate, actual costs diverge from it in both directions:

• Often lower than estimated – on a stable, unchanging system prompt and tool set, the real cache-hit rate is frequently higher than 50%, since the same content keeps getting reused call after call.
• Sometimes higher than estimated – on very long calls, the growing conversation history dilutes how much of each request is actually cached content, since new (uncached) transcript keeps accumulating.

Caching helps most with content that stays identical across requests within a call, and ideally across calls:

• The system prompt
• Tool definitions
• Any other static context included in every request (reference material, fixed instructions, etc.)

Content that changes on every request — like the growing transcript itself — can't benefit from caching in the same way, which is part of why long calls see less benefit than short ones.

The single biggest lever on both cost and latency is the size of your system prompt and tool definitions. Shortening them reduces the tokens processed on every single request, which has the largest effect on model cost and also reduces the time the model spends processing before it can start responding.

Flush syntax is a special token you can place in an assistant's output to force immediate transmission of the text before it to the voice provider, rather than waiting for normal buffering. This is especially useful for giving the caller something to hear right away during a long tool call.

Supported syntax (all case-insensitive):

<flush />
<flush>
</flush>

Configuration requirement: flush syntax only works when chunkPlan.enabled is true in your voice configuration. If it's false, the flush tags will be spoken aloud instead of being processed.

{
  "voice": {
    "chunkPlan": {
      "enabled": true
    }
  }
}

Example usage in a system prompt or response:

"Looking up that information... <flush /> This may take a moment."

When to avoid it: on every response (fragments the audio), mid-sentence (breaks natural speech flow), on short responses (normal buffering is already fine), or more than once per response (creates choppy audio). Frequent use also increases the number of API calls made to the voice provider, which can raise usage-based costs — use it strategically, not by default.

When a tool call is going to take a moment, you can configure a request-start message on that tool so an acknowledgment plays the instant the tool call begins, rather than waiting on the model to generate a spoken response first. This gets an acknowledgment out to the caller with no added LLM latency, since the message is played directly rather than generated:

"messages": [
  {
    "type": "request-start",
    "content": "One moment while I look that up for you."
  }
]

Combining a short, static request-start message with flush syntax for anything that follows gives callers immediate feedback on both fixed-wording acknowledgments and dynamically generated ones.