Voice Pipeline: Endpointing & Turn Detection

Overview

Endpointing is how your voice assistant decides that a caller has finished talking and it's time to respond. Get this wrong and the assistant either cuts people off mid-sentence or leaves awkward dead air. This page covers the full voice pipeline flow, the priority order Sulus uses to make that decision, and every configuration layer available to tune it.

This is the upstream turn-detection layer — it decides when the assistant should start talking. It is a different layer from the Interruption Handling & Barge-In Sensitivity page, which covers the downstream stopSpeakingPlan: what happens once the assistant is already speaking and the caller talks over it.

Every turn in a call moves through the same sequence of stages:

User Audio → VAD → Transcription → Start Speaking Decision → LLM → TTS → waitSeconds → Assistant Audio

The "Start Speaking Decision" step is where all of the endpointing configuration described below applies.

When more than one endpointing mechanism is configured, Sulus evaluates them in a fixed priority order:

1. Transcriber-level end-of-turn (EOT) detection — if your transcriber has built-in EOT detection (Deepgram Flux, AssemblyAI) and no smartEndpointingPlan is set, the transcriber's own signal is used.
2. Custom endpointing rules (customEndpointingRules) — the highest-priority override. If a configured regex rule matches the latest assistant or user message, its timeoutSeconds value is applied regardless of what any other plan would have decided.
3. Smart endpointing plan (smartEndpointingPlan) — a provider-driven confidence model (see below), used when no custom rule matches.

If neither a smart endpointing plan nor a transcriber with built-in EOT is configured, Sulus falls back to the rule-based transcriptionEndpointingPlan described further down this page.

Important: don't set a smartEndpointingPlan at the same time as a transcriber with built-in EOT detection (Deepgram Flux or AssemblyAI) — doing so bypasses the transcriber's own EOT events instead of combining with them.

customEndpointingRules let you extend the wait time in specific conversational contexts — for example, giving callers more time while spelling an email address or reading back a phone number.

{
  "customEndpointingRules": [
    {
      "type": "assistant",
      "regex": "(phone|email|address)",
      "timeoutSeconds": 3.0
    },
    {
      "type": "user",
      "regex": "\\d{3}-\\d{3}-\\d{4}",
      "timeoutSeconds": 2.0
    }
  ]
}

Good use cases: data collection (phone numbers, emails, addresses), spelling, and any moment where a caller is likely to pause mid-thought while providing detailed information.

The smartEndpointingPlan supports two provider types:

LiveKit (text-based, English)

LiveKit is a text-based provider recommended for English conversations. It uses a waitFunction — a mathematical expression that takes a confidence value (x, ranging 0–1) and returns a wait time in milliseconds before the assistant responds. Three preset styles are available:

{
  "startSpeakingPlan": {
    "smartEndpointingPlan": {
      "provider": "livekit",
      "waitFunction": "2000 / (1 + exp(-10 * (x - 0.5)))"
    }
  }
}

For non-English languages, use Sulus's own built-in smart endpointing provider instead of LiveKit:

{
  "startSpeakingPlan": {
    "smartEndpointingPlan": {
      "provider": "sulus"
    }
  }
}

Krisp (audio-based)

Krisp is an audio-based provider that analyzes prosodic features (intonation, pitch, rhythm) rather than transcript text, and works for non-English languages as well. It returns a probability between 0 and 1 via a threshold parameter, where 1 means the caller has definitely stopped speaking.

{
  "startSpeakingPlan": {
    "smartEndpointingPlan": {
      "provider": "krisp",
      "threshold": 0.5
    }
  }
}

Because Krisp is audio-based, it notifies the pipeline of any detected speech, including brief acknowledgments. Pair it with a well-configured stopSpeakingPlan (acknowledgementPhrases, numWords) so backchanneling like "okay" or "mhm" doesn't get treated as a full turn.

Some transcribers have their own built-in end-of-turn detection, configured directly on the transcriber object rather than through smartEndpointingPlan.

Deepgram Flux

Deepgram's Flux models detect end-of-turn using an eotThreshold confidence value (0–1, default 0.7) and an eotTimeoutMs maximum wait (default 5000ms).

{
  "transcriber": {
    "provider": "deepgram",
    "model": "flux-general-en",
    "language": "en",
    "eotThreshold": 0.7,
    "eotTimeoutMs": 5000
  }
}

A multilingual Flux model (flux-general-multi) is also available, supporting English, Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian, and Dutch — set language to one of these codes, or omit it for automatic detection.

Do not set a smartEndpointingPlan alongside Deepgram Flux — it bypasses Flux's own EOT events.

AssemblyAI

AssemblyAI's neural turn detection is configured with three transcriber-level fields:

As with Deepgram Flux, don't set a smartEndpointingPlan when using AssemblyAI's built-in turn detection.

When neither a smart endpointing plan nor a transcriber with built-in EOT is configured, Sulus falls back to transcriptionEndpointingPlan, which decides wait time purely from patterns in the transcribed text.

{
  "startSpeakingPlan": {
    "transcriptionEndpointingPlan": {
      "onPunctuationSeconds": 0.1,
      "onNoPunctuationSeconds": 1.5,
      "onNumberSeconds": 0.5
    },
    "waitSeconds": 0.4
  }
}

The system applies the first matching rule, in this priority order: a message ending in a number → onNumberSeconds; otherwise a message containing punctuation → onPunctuationSeconds; otherwise no punctuation detected → onNoPunctuationSeconds; otherwise, if nothing matches, the assistant responds immediately.

This plan is best used for non-English languages where smart endpointing isn't supported, as a fallback when smart endpointing is unavailable, or when you need predictable, rule-based timing. Note that a matching customEndpointingRules entry still overrides this plan.

waitSeconds is a final audio delay applied after all voice pipeline processing completes (endpointing → LLM → TTS), just before the assistant begins speaking. It's a separate knob from every endpointing mechanism above — it doesn't affect when the caller is judged to be done talking, only how long the assistant pauses once its response is ready.

Default is 0.4 seconds, which suits most standard conversations. Valid range is 0 to 5 seconds. Lower values feel snappier; higher values create a more patient, human-like pause — useful in sensitive or complex conversation contexts.

Example timeline with the default 0.4s value:

0.0s: User stops speaking
0.1s: Smart endpointing evaluation begins
0.6s: Smart endpointing triggers
0.6s: LLM request sent
1.4s: LLM response received (~0.8s processing)
1.9s: TTS audio generated (~0.5s processing)
1.9s: waitSeconds (0.4s) starts
2.3s: Assistant begins speaking