WebSocket Transport: Connection, Audio Format & Background Messages

Overview

WebSocket transport lets you open a real-time, bidirectional audio connection to an assistant instead of using a phone number or the web SDK. This page covers creating a call on this transport, connecting to the socket it returns, the two message types you'll see on that socket, and how background messages fit into this specific transport context.

The audio format requirements here are separate from — and looser than — the strict raw-PCM requirements that apply specifically to Custom Voice (TTS) webhook responses; don't confuse the two. For the general background-message concept across transports, see Async Server Tools & Background Messages, which this page applies specifically to the WebSocket transport context.

Create a call with transport.provider set to the WebSocket transport, and an audioFormat object describing how you want audio encoded:

curl 'https://api.sulus.ai/call' \
  -H 'authorization: Bearer YOUR_API_KEY' \
  -H 'content-type: application/json' \
  --data-raw '{
    "assistantId": "YOUR_ASSISTANT_ID",
    "transport": {
      "provider": "sulus.websocket",
      "audioFormat": {
        "format": "pcm_s16le",
        "container": "raw",
        "sampleRate": 16000
      }
    }
  }'

Sample rate conversion is automatic, so you can stream at any rate — 8kHz, 44.1kHz, or otherwise — and it will be converted as needed.

The call-creation response includes a websocketCallUrl inside the transport object:

{
  "id": "7420f27a-30fd-4f49-a995-5549ae7cc00d",
  "type": "sulus.websocketCall",
  "status": "queued",
  "transport": {
    "provider": "sulus.websocket",
    "websocketCallUrl": "wss://api.sulus.ai/7420f27a-30fd-4f49-a995-5549ae7cc00d/transport"
  }
}

Open the connection and wire up the standard handlers:

const socket = new WebSocket("wss://api.sulus.ai/7420f27a-30fd-4f49-a995-5549ae7cc00d/transport");

socket.onopen = () => console.log("WebSocket connection opened.");
socket.onclose = () => console.log("WebSocket connection closed.");
socket.onerror = (error) => console.error("WebSocket error:", error);

Note that when using WebSocket transport, phone-based parameters (phoneNumber or phoneNumberId) aren't permitted on the same call — the two are mutually exclusive.

Once open, the socket carries two kinds of messages: binary audio data (raw PCM Int16Array samples, or 8-bit Mu-Law samples, matching your configured audioFormat) and text-based JSON control messages.

Sending an audio chunk:

function sendAudioChunk(audioBuffer) {
  if (socket.readyState === WebSocket.OPEN) {
    socket.send(audioBuffer); // raw binary buffer
  }
}

navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
  const audioContext = new AudioContext();
  const source = audioContext.createMediaStreamSource(stream);
  const processor = audioContext.createScriptProcessor(1024, 1, 1);

  processor.onaudioprocess = (event) => {
    const pcmData = event.inputBuffer.getChannelData(0);
    const int16Data = new Int16Array(pcmData.length);

    for (let i = 0; i < pcmData.length; i++) {
      int16Data[i] = Math.max(-32768, Math.min(32767, pcmData[i] * 32768));
    }

    sendAudioChunk(int16Data.buffer);
  };

  source.connect(processor);
  processor.connect(audioContext.destination);
});

Receiving messages — distinguish binary audio blobs from JSON control messages:

socket.onmessage = (event) => {
  if (event.data instanceof Blob) {
    // Binary: assistant audio to play back
    event.data.arrayBuffer().then(buffer => {
      const audioData = new Int16Array(buffer);
      playAudio(audioData);
    });
  } else {
    // Text: JSON control/event message
    try {
      const message = JSON.parse(event.data);
      handleControlMessage(message);
    } catch (error) {
      console.error("Failed to parse message:", error);
    }
  }
};

On this transport, background messages are sent as JSON text messages directly over the same socket you're already using for audio — there's no separate endpoint to call. Use the add-message type to silently insert an item into the conversation history, for example a system-role message, without interrupting the audio stream:

function sendControlMessage(messageObj) {
  if (socket.readyState === WebSocket.OPEN) {
    socket.send(JSON.stringify(messageObj));
  }
}

sendControlMessage({
  type: "add-message",
  message: {
    role: "system",
    content: "The user has pressed the button, say peanuts",
  },
});

Accepted role values are system, user, assistant, tool, and function; system is recommended for anything that shouldn't notify the user.

Don't confuse this with the separate mechanism on Live Call Monitoring: that page covers an external supervisor injecting an add-message via an HTTP POST to a distinct controlUrl, for monitoring a call this transport's own participant is already on. Here, you're the call's own client, sending the same add-message type as a JSON message over the one active connection you already hold.