Simulations: Voice-Based AI-to-AI Testing

Overview

Simulations is Sulus's voice-based testing framework. Where Evals validates your assistant using scripted mock chat conversations (chat.mockConversation), Simulations tests your voice agent end-to-end over an actual voice call. An AI caller persona, with its own model and voice, calls your assistant and follows scenario instructions, and the full audio conversation is recorded, transcribed, and scored against criteria you define.

A simulation runs in four stages:

1. Simulation — an AI tester with a defined personality calls your voice agent and follows scenario instructions
2. Conversation — both AIs engage in a natural voice conversation
3. Recording — the entire call is recorded and transcribed
4. Assessment — the transcript is evaluated against your defined criteria by an LLM

The workflow has five steps:

1. Create a personality — define the AI tester's voice, model, and behavior via a system prompt (e.g. "Impatient Customer", "Confused User")
2. Create a scenario — specify what the tester should do and define evaluations using structured outputs
3. Create a simulation — pair a scenario with a personality
4. Run the simulation — execute against your assistant or squad in voice or chat mode
5. Review results — analyze pass/fail outcomes based on structured output evaluations

Example: creating a personality via the API:

curl -X POST "https://api.sulus.ai/eval/simulation/personality" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Impatient Customer",
    "assistant": {
      "model": {
        "provider": "openai",
        "model": "gpt-4o",
        "messages": [
          {
            "role": "system",
            "content": "You are an impatient customer who wants quick answers. Speak directly and may interrupt if responses are too long."
          }
        ]
      },
      "voice": {
        "provider": "cartesia",
        "voiceId": "sonic-english"
      }
    }
  }'

Example: creating a scenario with a structured-output evaluation:

curl -X POST "https://api.sulus.ai/eval/simulation/scenario" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Book Appointment",
    "instructions": "You are calling to book an appointment for next Monday at 2pm. End the call once you receive a confirmation number.",
    "evaluations": [
      {
        "structuredOutput": {
          "name": "appointment_booked",
          "schema": { "type": "boolean", "description": "Whether an appointment was successfully booked" }
        },
        "comparator": "=",
        "value": true,
        "required": true
      }
    ]
  }'

Each scenario evaluation supports these comparators:

Choose Simulations instead of (or in addition to) Evals when you need to validate anything that only shows up over real audio, such as:

• Speech-to-text transcription accuracy for your specific vocabulary and accents
• Interruption and barge-in handling
• Hooks and webhook behavior triggered by call lifecycle events
• Squad handoff testing over an actual call
• Final pre-deployment validation before a configuration goes live

Evals (chat.mockConversation) remains the faster, cheaper option for validating conversation logic, tool-calling arguments, and prompt behavior without the cost of audio processing.

Keep in mind that voice-mode simulations take more time to execute than chat-mode tests, each test consumes calling minutes from your account, and maximum call duration is 15 minutes per test.

Simulation hooks let you trigger webhook notifications on lifecycle events, making it straightforward to wire test results into a CI/CD pipeline. Hooks are only supported in voice mode and require the websocket transport — they will not trigger in chat mode.

Supported events:

• simulation.run.started — fires when a simulation run begins
• simulation.run.ended — fires when a simulation run ends, and includes pass/fail status

Example scenario with a CI/CD hook:

curl -X POST "https://api.sulus.ai/eval/simulation/scenario" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Regression Suite with CI Hooks",
    "instructions": "Complete the booking flow as a standard customer.",
    "evaluations": [...],
    "hooks": [
      {
        "on": "simulation.run.ended",
        "do": [
          {
            "type": "webhook",
            "server": {
              "url": "https://your-ci-server.com/webhooks/simulation-ended"
            },
            "include": {
              "transcript": true,
              "messages": true,
              "recordingUrl": true
            }
          }
        ]
      }
    ]
  }'

The simulation.run.ended webhook payload includes a status field (passed or failed) that your CI/CD system can use as a quality gate to block or allow a deployment. A recommended pipeline order is: fast smoke tests first, then named regression tests, then edge case tests, then the full simulation suite as the final gate before release.

In summary, Simulations complements Evals by testing your voice agent the way a real caller would experience it — over actual audio, with real transcription and real timing — and its webhook hooks make it straightforward to fold into an automated regression pipeline.