Evals

Score prompt versions, labels and A/B experiments against a dataset with an LLM judge.

Evals answer "is this prompt version actually better?" with numbers. You keep a dataset of test rows, start a run against a managed prompt (Prompts) — a pinned version, a label such as prod, or a whole A/B experiment — and a judge model scores every output against your rubric. Results sit beside the cost and latency of the same calls, so a quality win can be weighed against what it costs.

Everything runs through the gateway: the target-model call and the judge call are billed, cached, rate-limited and logged exactly like your own traffic, and are tagged x-ai-setu-metadata: {"source":"evals"} so you can split eval spend from production spend in the console's cost views.

Concepts

Thing	What it is
Dataset	A named set of up to 2,000 rows in a workspace.
Row	Template `variables`, optional extra chat `messages` appended after the rendered prompt, an optional `expected` reference answer, and `tags`.
Run	One evaluation: a target (version / label / experiment), a judge model, a rubric, a score scale and a pass threshold.
Result	One (row, variant) outcome: the model output, the judge's score + reasoning, pass/fail, latency, cost and the gateway request ids.

Targets. A version run evaluates exactly that version. A label run is pinned to the version the label points at when the run starts (moving the label later does not change a running eval). An experiment run evaluates every variant version on every row, so each variant gets its own mean score, pass rate, cost and latency — the A/B comparison on quality, not just on traffic split.

Scales. ONE_TO_FIVE (integer 1–5) or ZERO_TO_ONE (0–1). A result passes when score >= passThreshold. Pass rate counts errored rows as failures.

Judge. Defaults to @mantle-openai/openai.gpt-5.6-terra. The judge is asked for {"score", "reasoning"} JSON using strict structured output; if the judge model rejects response_format, the run falls back to a plain JSON instruction and lenient parsing. Any model your workspace can route works as the judge.

Worked example — A/B two prompt versions

You have a template support-reply with an experiment tone-test splitting traffic between v2 (current) and v3 (friendlier). Score both on the same 3 tickets.

1. Create a dataset and paste rows (console: Evals → Datasets, or GraphQL). CSV columns become template variables; expected and tags are special:

ticket,expected,tags
"My order #812 hasn't arrived",Apologise and give the tracking link,shipping
"How do I reset my password?",Point to Settings → Security → Reset,account
"Cancel my subscription",Confirm cancellation steps without upselling,billing

JSON works too — [{"variables": {"ticket": "…"}, "expected": "…"}] or one object per line.

mutation {
  createEvalDataset(input: { workspaceId: "<workspace-id>", name: "Support tickets" }) {
    id
  }
}
mutation {
  importEvalRows(datasetId: "<dataset-id>", text: "ticket,expected\n…")
}

2. Start a run (console: Evals → Start run). Pick the API key the run is billed to — it must belong to the same workspace and have chat:write.

mutation {
  startEvalRun(
    input: {
      datasetId: "<dataset-id>"
      templateId: "<support-reply-id>"
      experimentId: "<tone-test-id>"
      apiKeyId: "<api-key-id>"
      rubric: "The reply must resolve the ticket, match the expected action, and be polite and concise."
      scale: ONE_TO_FIVE
      passThreshold: 4
    }
  ) {
    id
    status
    resolvedVersions
  }
}

3. Read the per-variant summary once the run completes (the console polls while it runs):

query {
  evalRun(id: "<run-id>") {
    status
    stats {
      mean
      passRate
      costMicros
      variants {
        variant
        mean
        passRate
        costMicros
        latencyP50
      }
    }
  }
}

{
  "status": "completed",
  "stats": {
    "mean": 4.17,
    "passRate": 0.833,
    "costMicros": "4210",
    "variants": [
      { "variant": 2, "mean": 3.67, "passRate": 0.667, "costMicros": "2050", "latencyP50": 812 },
      { "variant": 3, "mean": 4.67, "passRate": 1, "costMicros": "2160", "latencyP50": 840 }
    ]
  }
}

v3 scores a point higher for ~5% more cost and similar latency — promote it with the prod label. Drill into evalResults(runId, variant) for each row's output and the judge's reasoning.

costMicros fills in a minute or so after the run ends (it is read from the billing ledger); until then costPending counts the rows still waiting.

Limits

2,000 rows per dataset (and so per run); 500 rows per import or addEvalRows call.
A run stops after 30 minutes of wall-clock time and is marked failed with the rows scored so far kept.
Four rows are evaluated concurrently per run. Gateway 429/5xx are retried with backoff; 401/402/403 (for example insufficient credit) stop the run.
cancelEvalRun stops a queued or running run; rows already scored are kept.

Permissions and security

Reading evals needs settings view; creating datasets, starting and cancelling runs needs settings update (the same gates as prompt management). A run never sees a raw API key: the worker mints a short-lived, chat-scoped ephemeral token from the key you pick, uses it for the run's gateway calls, and revokes it when the run ends. Dataset rows and model outputs are stored in your tenant like other prompt data. Treat the judge's score as advisory: model outputs being judged are placed in delimited blocks the judge is told to treat as data, but a judge model can still be influenced by what it reads.

Concepts

Worked example — A/B two prompt versions

Limits

Permissions and security

On this page