Assistant Testing (Evals)

Overview

Sulus's Evals is an agent testing framework that lets you systematically test assistants and squads using mock conversations with automated validation. It's the recommended way to catch regressions before they reach live callers.

• Creating mock conversations — define user messages and expected assistant responses
• Validating behavior — use exact match, regex patterns, or AI-powered judging
• Testing tool calls — verify function calls with specific arguments
• Running automated tests — execute tests and receive detailed pass/fail results
• Debugging failures — review full conversation transcripts with evaluation details

Send a POST request to /eval with your mock conversation:

curl -X POST "https://api.sulus.ai/eval" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Greeting Test",
    "description": "Verify assistant greets users appropriately",
    "type": "chat.mockConversation",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      },
      {
        "role": "assistant",
        "judgePlan": {
          "type": "exact",
          "content": "Hello! How can I help you today?"
        }
      }
    ]
  }'

Save the returned id — you'll need it to run the evaluation.

Send a POST request to /eval/run targeting your assistant or squad:

curl -X POST "https://api.sulus.ai/eval/run" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evalId": "550e8400-e29b-41d4-a716-446655440000",
    "target": {
      "type": "assistant",
      "assistantId": "your-assistant-id"
    }
  }'

Then poll for results:

curl -X GET "https://api.sulus.ai/eval/run/eval-run-123" \
  -H "Authorization: Bearer $SULUS_API_KEY"

A passing evaluation returns:

• status: "ended"
• endedReason: "mockConversation.done"
• results[0].status: "pass"
• All judge.status values: "pass"

A failing evaluation includes a judge.failureReason explaining what went wrong.

Choose the right judge type for each scenario:

Validate that your assistant calls functions with the correct arguments:

{
  "judgePlan": {
    "type": "exact",
    "toolCalls": [
      {
        "name": "bookAppointment",
        "arguments": {
          "date": "2025-01-20",
          "time": "14:00"
        }
      }
    ]
  }
}

Tool calls are validated in the order they're defined.

For production-grade test suites, consider these strategies:

• Smoke tests — quick 1–2 turn checks to verify basic response before running full suites
• Regression tests — named after bugs they prevent (e.g., "Regression: Date Parsing Bug #1234")
• Edge case tests — empty input, special characters, ambiguous requests, topic changes
• Happy path tests — validate ideal user journeys end-to-end
• Error handling tests — simulate tool failures, timeouts, and invalid inputs

In summary, Sulus's Evals framework supports the full testing lifecycle — from simple greeting checks to complex multi-turn flows with tool call validation and AI-powered semantic judging. You can also integrate evals into CI/CD pipelines as automated quality gates before deployment.