Test Suites & Evals: Regression Testing

Overview

This page covers regression-testing methodology across Sulus's testing tools, and the Test-Suites-vs-Evals/Simulations landscape. It does not repeat the basic API mechanics already covered on Assistant Testing (Evals), Eval API, or Scorecard: Automated Call Grading — see those pages for endpoint-by-endpoint reference.

Test Suites: Rubric-Based Evaluation

Test Suites automate end-to-end testing by having an AI tester interact with your voice agent following a pre-defined script. After the interaction, the full conversation transcript and your evaluation rubric (one or more questions) are sent to an LLM, which determines whether the interaction met the defined objectives.

Key configuration options per test case:

Script – defines how the AI tester should behave, step-by-step or free-form
Type – Chat (faster, text-based) or Voice (full call simulation with recording)
Rubric – one or more questions the LLM uses to judge success
Attempts – up to 5 runs per test case per suite execution, up to 50 test cases per suite

Test calls are not free — they cost the same as regular calls.

Direction: Moving Toward Simulations

Test Suites is being deprecated in favor of Simulations, which replaces LLM rubric scoring with structured-output evaluations and AI-powered caller personalities. No specific deprecation date has been confirmed, but the direction is clear: new regression-testing work should be built on Simulations (for voice-based testing) and Evals (for chat-based mock conversations) rather than on Test Suites going forward.

The Three Judge/Scorecard Types

Both Evals and Simulations validate a response using one of three approaches. Picking the right one for a given check matters as much as writing the test itself:

Judge Type	Best For
`exact`	Critical business data, compliance-required wording, tool call arguments – anything that must match precisely
`regex`	Variable responses such as names, dates, or IDs, and flexible phrasing that still needs to fit a pattern
`ai`	Semantic meaning, tone or sentiment, and contextual appropriateness — cases where there's no single correct string

As a rule of thumb: reach for exact first when you can, since it's the least likely to produce a false pass; fall back to regex when the exact wording will legitimately vary; and use ai only when you're checking something genuinely semantic that neither of the other two can express.

Regression-Testing Best Practices

Name with a "Regression: " prefix — e.g. "Regression: Date Parsing Bug #1234" — so a regression test is identifiable at a glance
Include the ticket/issue number in both the name and the description, so the test is traceable back to what it prevents
Add a regression test whenever a bug is fixed — this is what keeps the suite growing in step with real incidents rather than by guesswork
Run the full suite before major releases, not just the newest tests
Archive rather than delete tests for removed features — don't delete a regression test just because the bug seems old; only archive it once the feature it covers is fully removed

A recommended pipeline order when running tests as part of a release process: fast smoke tests first, then named regression tests, then edge case tests, then the full suite as the final gate before release.

Worked Regression Test Example

Chat-based (Evals), verifying a date-parsing bug fix:

curl -X POST "https://api.sulus.ai/eval" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Regression: Date Parsing Bug #1234",
    "description": "Verify dates like 3/15 are parsed correctly after fix",
    "type": "chat.mockConversation",
    "messages": [
      { "role": "user", "content": "Book me for 3/15" },
      {
        "role": "assistant",
        "judgePlan": {
          "type": "exact",
          "toolCalls": [{
            "name": "bookAppointment",
            "arguments": { "date": "2026-03-15" }
          }]
        }
      }
    ]
  }'

The equivalent voice-based (Simulations) scenario for the same bug:

curl -X POST "https://api.sulus.ai/eval/simulation/scenario" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Regression: Date Parsing Bug #1234",
    "instructions": "Request an appointment for 3/15. The assistant should correctly parse this as March 15th, not fail or misinterpret the date.",
    "evaluations": [
      {
        "structuredOutput": {
          "name": "date_parsed_correctly",
          "schema": { "type": "boolean", "description": "Whether 3/15 was correctly understood as March 15th" }
        },
        "comparator": "=",
        "value": true
      }
    ]
  }'

Summary

Test Suites still work today but are on their way out in favor of Simulations; either way, the same regression discipline applies — name tests after the bug they prevent, include the ticket number, add a test the moment you fix something, run the full suite before releases, and archive (never delete) tests for removed features. For the underlying API mechanics, see Assistant Testing (Evals) and Eval API; for grading live call quality after the fact rather than in a test run, see Scorecard: Automated Call Grading.

On this page

Test Suites & Evals: Regression Testing