Custom LLM Tool Calling & Streaming

Overview

The Custom LLM (Bring Your Own Model Endpoint) page covers the basic setup for pointing an assistant at your own OpenAI-compatible endpoint. This page goes further into the mechanics you need once that assistant also needs to call tools — the three distinct tool-calling patterns a custom LLM endpoint must support, how streaming responses work with each, and the exact response formats expected back from your server.

Your custom LLM's /chat/completions endpoint must accept the incoming payload and stream its response back as Server-Sent Events (SSE):

app.post("/chat/completions", async (req, res) => {
  const payload = req.body;
  const requestArgs = {
    model: payload.model,
    messages: payload.messages,
    temperature: payload.temperature ?? 1.0,
    stream: true,
    tools: payload.tools || [],
    tool_choice: "auto",
  };

  const llmResponse = await openai.chat.completions.create(requestArgs);

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");

  for await (const chunk of llmResponse) {
    res.write(`data: ${JSON.stringify(chunk)}\n\n`);
  }
  res.write("data: [DONE]\n\n");
  res.end();
});

The incoming tools array in the payload reflects whatever tools are attached to the assistant — your server passes it through to the underlying model so the model knows what's available to call.

Native LLM tools are functions your custom LLM server executes itself (for example, get_payment_link). The flow is:

1. The model's streaming response includes a tool call in its chunks
2. Your server detects finish_reason === "tool_calls", parses the accumulated arguments, and executes the function directly
3. The result is sent back to the model as a follow-up message, and that follow-up response is streamed to the assistant the same way as the original response

const finishReason = choice?.finish_reason;
if (finishReason === "tool_calls" && toolCallInfo) {
  const result = await tool_functions[toolCallInfo.name](parsedArgs);
  const functionMessage = {
    role: "function",
    name: toolCallInfo.name,
    content: JSON.stringify(result)
  };
  const followUpResponse = await openai.chat.completions.create({
    model: requestArgs.model,
    messages: [...requestArgs.messages, functionMessage],
    stream: true,
    tools: requestArgs.tools,
    tool_choice: "auto"
  });
  for await (const followUpChunk of followUpResponse) {
    res.write(`data: ${JSON.stringify(followUpChunk)}\n\n`);
  }
}

Sulus-attached tools are pre-configured directly on the assistant (for example, transferCall). When your server detects one of these tool calls in the model's output, it should not execute the function itself — instead, write a function_call payload directly to the stream and let the platform handle execution:

if (functionName === "transferCall" && payload.destination) {
  const functionCallPayload = {
    function_call: {
      name: "transferCall",
      arguments: { destination: payload.destination },
    },
  };
  res.write(`data: ${JSON.stringify(functionCallPayload)}\n\n`);
  continue;
}

The distinction from Native LLM Tools matters: a Sulus-attached tool's execution logic lives on the platform side, not in your server code, so your server's job is only to recognize the call and hand it back — not to run it.

Custom tools are handled by a separate, dedicated endpoint on your server (for example, /chat/completions/custom-tool) rather than inline in the main completions stream. The platform sends the tool call list to that endpoint, and your server responds with a JSON results array:

app.post("/chat/completions/custom-tool", async (req, res) => {
  const requestPayload = req.body.message;
  for (const toolCall of requestPayload.toolCallList) {
    if (toolCall.function?.name === "processOrder") {
      return res.json({
        results: [{ toolCallId: toolCall.id, result: "Order processed successfully" }],
      });
    }
  }
});

Note the response shape difference: this endpoint returns a single JSON object with a results array (not an SSE stream), matching each result back to its toolCallId.

Attach both platform tools and custom tools in the same assistant PATCH request. Here's an example combining a transferCall (Sulus-attached) tool with a processOrder (custom, dedicated-endpoint) tool:

curl -X PATCH https://api.sulus.ai/assistant/YOUR_ASSISTANT_ID \
     -H "Authorization: Bearer YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
  "model": {
    "provider": "custom-llm",
    "model": "gpt-4o",
    "url": "https://your-custom-llm-url/chat/completions",
    "tools": [
      {
        "type": "transferCall",
        "destinations": [
          {
            "type": "number",
            "number": "+15551234567",
            "numberE164CheckEnabled": false,
            "message": "Transferring you to our customer service department."
          }
        ]
      },
      {
        "type": "function",
        "async": false,
        "function": {
          "name": "processOrder",
          "description": "Processes a customer order"
        },
        "server": {
          "url": "https://your-custom-llm-url/chat/completions/custom-tool"
        }
      }
    ]
  }
}'

For the base setup steps — registering credentials and creating the assistant with the custom-llm provider — see the Custom LLM (Bring Your Own Model Endpoint) page.