Scorecard: Automated Call Grading

Overview

A scorecard automatically grades a call against KPIs you define, right after the call ends. It evaluates one or more structured outputs against conditions you set, allocates points for each condition that's met, and produces a single score you can use for QA dashboards, filtering, or triggering follow-up automation.

Scorecards are built on top of structured outputs, so you'll need those set up first — see Structured Outputs (Post-Call Data Extraction) before continuing here.

Before creating a scorecard, you need:

• An assistant with a model and voice already configured
• One or more structured outputs created and attached to that assistant via artifactPlan.structuredOutputIds

Scorecard metrics can only reference structured outputs of type number/integer or boolean. A structured output field of type string or object cannot be used as a scorecard metric.

Create a scorecard with the observability endpoint, defining one metric per structured output you want to grade against:

curl -X POST https://api.sulus.ai/observability/scorecard \
  -H "Authorization: Bearer $CORE_SYSTEM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support QA Scorecard",
    "description": "Grades calls based on resolution and satisfaction",
    "metrics": [
      {
        "structuredOutputId": "UUID-OF-BOOLEAN-STRUCTURED-OUTPUT",
        "conditions": [
          { "type": "comparator", "comparator": "=", "value": true, "points": 70 }
        ]
      },
      {
        "structuredOutputId": "UUID-OF-NUMBER-STRUCTURED-OUTPUT",
        "conditions": [
          { "type": "comparator", "comparator": ">=", "value": 8, "points": 30 }
        ]
      }
    ],
    "assistantIds": ["your-assistant-id"]
  }'

Rules for metrics and conditions:

• Supported comparators: =, !=, >, <, >=, <=
• Boolean metrics only support the = comparator
• Points across all conditions must sum to exactly 100
• Each condition's points must be between 0 and 100

Link a saved scorecard to an assistant either by passing assistantIds when you create it, or by updating the assistant directly:

curl -X PATCH https://api.sulus.ai/assistant/YOUR_ASSISTANT_ID \
  -H "Authorization: Bearer $CORE_SYSTEM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "artifactPlan": {
      "structuredOutputIds": ["UUID-OF-BOOLEAN-STRUCTURED-OUTPUT", "UUID-OF-NUMBER-STRUCTURED-OUTPUT"],
      "scorecardIds": ["YOUR_SCORECARD_ID"]
    }
  }'

After a call ends, fetch the call and read call.artifact.scorecards:

{
  "c5bd0d4e-0c7f-4b0a-82e9-5d0baf6e4a01": {
    "name": "Support QA Scorecard",
    "score": 70,
    "scoreNormalized": 70,
    "metricPoints": {
      "UUID-OF-BOOLEAN-STRUCTURED-OUTPUT": 70,
      "UUID-OF-NUMBER-STRUCTURED-OUTPUT": 0
    }
  }
}

For quick testing without saving a scorecard as its own resource, embed one inline using artifactPlan.scorecards instead of scorecardIds:

curl -X PATCH https://api.sulus.ai/assistant/YOUR_ASSISTANT_ID \
  -H "Authorization: Bearer $CORE_SYSTEM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "artifactPlan": {
      "structuredOutputIds": ["SO-UUID-BOOLEAN", "SO-UUID-NUMBER"],
      "scorecards": [
        { "name": "Inline Test Scorecard", "metrics": [ ] }
      ]
    }
  }'

In summary, a scorecard turns your structured output values into a single, consistent grade per call — making it straightforward to power QA dashboards or filter calls by quality without manually reviewing every transcript.