Orchestration Layer

Overview

Underneath every voice call, the core system runs an orchestration layer — a set of proprietary real-time models that sit alongside the transcriber, LLM, and voice components and make a conversation feel natural rather than mechanical. Unlike the transcriber, LLM, and voice layers, which are all bring-your-own-provider or bring-your-own-server capable, the orchestration layer runs exclusively on the core system's own infrastructure and cannot be swapped out or replaced with a custom server.

Endpointing and background noise filtering are covered in their own dedicated pages — see Voice Pipeline: Endpointing & Turn Detection and Background Denoising. This page focuses on the three orchestration models that shape how the assistant sounds and reacts: emotion detection, backchanneling, and filler injection.

Emotion detection analyzes the tone of the caller's speech in real time — for example, whether they sound frustrated, confused, or satisfied — and passes that reading to the LLM as ephemeral context alongside the transcript. The model can use this signal to adjust its response, such as slowing down and acknowledging frustration before continuing with a task, or matching a caller's upbeat tone.

Emotion metadata is not stored. It's generated and consumed in the moment to shape the very next response, then discarded — it does not appear in call logs, transcripts, or any other stored artifact.

Backchanneling is the natural habit of a listener giving small verbal signals while someone else is speaking — "uh-huh", "yeah", "got it", "okay" — without actually taking a turn in the conversation. Humans do this constantly and unconsciously; without it, an assistant that stays completely silent while listening can feel unresponsive or even seem like the line dropped.

The orchestration layer inserts these short affirmations at natural points while the caller is talking, so the conversation feels like it's being actively listened to. This is distinct from the assistant actually responding — a backchannel doesn't end the caller's turn or trigger a real reply from the LLM.

Filler injection adds natural speech disfluencies — "um", "like", "so", "you know" — into the assistant's own spoken responses. Real human speech is rarely perfectly clean; a small amount of natural filler can make an assistant's voice sound less scripted and more conversational, particularly during longer or more complex responses.

Two things distinguish the orchestration layer from every other configurable component of an assistant:

• Not customizable, no custom server option. The transcriber, LLM, and voice components can each be swapped for a different provider or replaced entirely with your own custom server (see Custom LLM and Custom Voice (TTS)). The orchestration layer has no equivalent bring-your-own option — emotion detection, backchanneling, and filler injection run exclusively on the core system's own infrastructure, and there is no configuration path to disable, replace, or relocate them.
• Ephemeral processing only. All orchestration-layer processing happens in real time and is not persisted. No audio, transcript excerpts, or emotion metadata generated by this layer are stored. Only the final call transcript and standard call logs are retained — the intermediate signals this layer generates to make the conversation feel natural are discarded as soon as they've been used.

This ephemeral, fixed-infrastructure behavior also has data-residency implications for regulated organizations — see the orchestration layer note in HIPAA Subprocessors & Data Residency for how it interacts with in-region custom storage.

In summary, the orchestration layer is the core system's own real-time conversational polish — endpointing, interruption detection, noise filtering, backchanneling, emotion detection, and filler injection working together. It's proprietary and cannot be customized or replaced, and every signal it generates is ephemeral, with only the final transcript and call logs ever stored.