Eval API: Create, Run, Get, List, Delete

Overview

Assistant Testing (Evals) covers the concepts and best-practice testing strategies behind the Evals framework. This page is the complete endpoint-by-endpoint API reference for the eval lifecycle — creating an eval definition, running it, checking a run's status, listing runs, and deleting a run.

Create an Eval

Send a POST request to /eval with a mock conversation. Each conversation turn has a role of user, assistant, system, or tool; assistant turns can carry a judgePlan describing what to validate.

curl -X POST "https://api.sulus.ai/eval" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Greeting Test",
    "type": "chat.mockConversation",
    "messages": [
      { "role": "user", "content": "Hello" },
      {
        "role": "assistant",
        "judgePlan": {
          "type": "exact",
          "content": "Hello! How can I help you today?"
        }
      }
    ]
  }'

There are three judgePlan types:

Type	Use Case
`exact`	Strict match on content or tool call arguments
`regex`	Flexible pattern matching for variable data (dates, names, IDs)
`ai`	Semantic/contextual evaluation via an LLM judge

Save the returned id — you'll need it to create a run.

Create a Run

Send a POST request to /eval/run with the evalId and a target object. The target can reference an existing assistantId or squadId, or pass a full transient assistant/squad object directly:

curl -X POST "https://api.sulus.ai/eval/run" \
  -H "Authorization: Bearer $SULUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evalId": "550e8400-e29b-41d4-a716-446655440000",
    "target": {
      "type": "assistant",
      "assistantId": "your-assistant-id"
    }
  }'

The response starts with status: "queued".

Get a Run

Poll GET /eval/run/{id} to check a run's progress and results:

curl -X GET "https://api.sulus.ai/eval/run/eval-run-123" \
  -H "Authorization: Bearer $SULUS_API_KEY"

A run's status is one of queued, running, or ended. Once ended, endedReason explains how it concluded — one of mockConversation.done, error, timeout, cancelled, or aborted.

A successful run has status: "ended", endedReason: "mockConversation.done", and every judge.status value equal to "pass". Any other endedReason, or a judge.status of "fail", means the run needs attention — check judge.failureReason for specifics on a failed judge.

List Runs

GET /eval/run returns a paginated list of eval runs, with these query parameters (all optional):

Parameter	Type	Description
`id`	string	Filter to a specific run ID.
`page`	number	Page number to return. Defaults to 1.
`limit`	number	Maximum number of items to return. Defaults to 100.
`sortOrder`	string	Sort order. Defaults to `DESC`. Valid values: `ASC`, `DESC`.
`createdAtGt` / `createdAtLt` / `createdAtGe` / `createdAtLe`	date-time	Filter by `createdAt` greater than / less than / greater-or-equal / less-or-equal a given timestamp.
`updatedAtGt` / `updatedAtLt` / `updatedAtGe` / `updatedAtLe`	date-time	Filter by `updatedAt` using the same four comparisons.

curl -X GET "https://api.sulus.ai/eval/run?page=1&limit=10&sortOrder=DESC" \
  -H "Authorization: Bearer $SULUS_API_KEY"

Delete a Run

DELETE /eval/run/{id} deletes the specified run and returns the deleted EvalRun object:

curl -X DELETE "https://api.sulus.ai/eval/run/{id}" \
  -H "Authorization: Bearer $SULUS_API_KEY"

CI/CD Pattern

The recommended pattern for wiring Evals into a CI/CD pipeline is to chain the endpoints above: create a run, then poll get run until status is ended, then check the result (endedReason and every judge.status) and fail the pipeline step if anything other than a clean pass comes back. This lets you gate a deployment on a passing eval suite the same way you'd gate it on any other automated test.

On this page

Eval API: Create, Run, Get, List, Delete