# Evaluation and experiments

> Tracon documentation. Tracon is a .NET package family that adds a control
> plane on Microsoft Agent Framework, published as NuGet packages that run
> inside your own ASP.NET Core application.
>
> Page: https://tracon.dev/concepts/evaluation/
> Summary: Improve agents with regression suites, production-run promotion, online judges, human feedback, and controlled experiments.

Use evaluation suites, judges, experiments, and regression checks to compare agent behavior against explicit criteria. They answer different questions
and are meant to be used together.

| | Question | When it runs |
|---|---|---|
| **Eval suites** | Did this change break anything? | On demand, against a fixed case set |
| **Online judges** | Is production drifting? | Continuously, on sampled live runs |
| **Human feedback** | What do people think? | Whenever someone scores a run |
| **Experiments** | Is version B better than A? | Live, splitting real traffic |

## Eval suites

A suite names the agent under test and carries **declarative checks** — a JSON array
stored and read as one unit with the suite. Create or update the suite before adding
cases:

```bash
curl -X PUT http://localhost:5081/tracon/api/evals/support \
     -H 'Content-Type: application/json' \
     -d '{
       "agentName": "support",
       "checks": [{"kind":"toolCalled","tools":["get_order_status"]}]
     }'
```

A suite needs at least one check before it can run — an empty `checks` array fails the
run outright instead of reporting every case as passed. Seven built-in kinds cover the
common cases; six match `Microsoft.Agents.AI.EvalChecks` factories directly and
`toolCallArgsMatch` is Tracon's own:

| Kind | Checks | Fields |
|---|---|---|
| `nonEmpty` | The response has at least `minLength` characters | `minLength` (default 1) |
| `containsExpected` | The response contains the case's `expectedOutput` | `caseSensitive` (default false) |
| `keywords` | The response contains every string in `values` | `values`, `caseSensitive` |
| `toolCalled` | The listed `tools` were called, `all` or `any` of them | `tools`, `mode` (default `all`) |
| `toolCallsPresent` | At least one tool was called | — |
| `toolCallArgsMatch` | Every tool call in the case's `expectedToolCalls` was made, with the listed arguments | — (reads the case) |
| `hasImageContent` | The response carries image content | — |

Application code can add more with `ITraconBuilder.AddEvalCheck(kind, check)` — a
named MAF `EvalCheck` that becomes usable under a custom kind name alongside the seven
built-in ones. A kind that matches neither fails the run with a clear error instead of
being silently skipped.

Cases are a separate, ordered list of queries:

```bash
curl -X PUT http://localhost:5081/tracon/api/evals/support/cases \
     -H 'Content-Type: application/json' \
     -d '[{"id":"01a0b449-ac16-712d-8bd8-1a60155d0d87",
          "query":"Where is order 4182?","expectedOutput":"shipped"},
         {"query":"Has order 4182 shipped yet?","expectedOutput":"shipped"}]'
```

That `PUT` is a **full replacement**: cases missing from the body are removed, so send
the whole list every time. Send each case you are keeping back with the `id` it was
listed with — a case holds that identifier for life, and runs are compared by it (see
below). A case that arrives without an `id` is a **new** case, so an unchanged case
sent without its id reads as one removed and another added; that is also how a real
regression in the same window escapes `--max-regressions`, which ignores removals by
design. An `id` that does not belong to this suite, or that appears twice, fails the
whole request with `400` and nothing is written. `parameters` travels back the same
way: it is part of the case, not of the request. Sequence numbers come from the body's
order, so reordering re-numbers the cases — but a sequence number is only display
order and does not change which case is which. `expectedOutput` reaches `containsExpected`.
The tool names a `toolCalled` check verifies come from the suite's own check definition,
shown above; the calls a `toolCallArgsMatch` check verifies come from the case itself.

A case's `context` is reference text for the agent. It reaches the agent through the
[document channel](/concepts/governance/#the-document-channel), ahead of the query —
the messages a run with `documents` sends. Plain text goes as one document named
`context`. Text made only of document blocks (`-----BEGIN TRACON DOCUMENT-----` …
`-----END TRACON DOCUMENT-----`), which is what a promoted run's documents become, goes
back as one document per block, unchanged. The evaluator receives the same text as the
case's context, for checks that judge an answer against reference material.

For an agent that declares [parameters](/concepts/agents/#parameters), a case's own
`parameters` field supplies the values that run's instructions bind against. A case
missing a value the agent requires fails outright, with a reason naming which
parameter is missing — the same check `POST /api/agents/{name}/run` applies, so a
case that would fail in production fails here too, before any model call is made.
The case run records the values it was bound with, defaults included, as every other
run does.

Running a suite queues a job. Each case runs in its own fresh session against the
agent and produces its own run row, so a failing check can be traced to the exact
conversation that produced it. That session is not stored, so a case whose run asks
for a tool approval fails with a reason that says so, and its run is closed as
`Failed`: nobody can answer the approval.

```bash
curl -X POST http://localhost:5081/tracon/api/evals/support/run
curl http://localhost:5081/tracon/api/evals/support/runs
```

### Checking which tool was called with which arguments

A case lists the tool calls it expects in `expectedToolCalls`. Each entry is a tool
`name` and, optionally, the `arguments` that call must carry:

```bash
curl -X PUT http://localhost:5081/tracon/api/evals/support \
     -H 'Content-Type: application/json' \
     -d '{"agentName":"support","checks":[{"kind":"toolCallArgsMatch"}]}'

curl -X PUT http://localhost:5081/tracon/api/evals/support/cases \
     -H 'Content-Type: application/json' \
     -d '[{"query":"Where is order ORD-1001?",
          "expectedToolCalls":[
            {"name":"get_order_status","arguments":{"orderId":"ORD-1001"}},
            {"name":"get_shipping_status"}]}]'
```

The case passes the check when, for every entry, the agent called a tool of that name
(letter case does not matter) and, when the entry lists `arguments`, one such call carried
every listed argument with an equal value. An entry without `arguments` checks only that
the tool was called. A failing result names each entry and what was wrong — `not called`
or `args mismatch` — without repeating the argument values the model sent.

- **The listed arguments are a subset.** The call may carry others, and an argument name
  matches in any letter case.
- **A value compares by its JSON meaning.** `2` equals `2.0`, an object equals an object
  with the same properties in any order, and spacing between tokens does not matter. A
  nested object or array must match exactly; the subset rule applies to the top-level
  argument names only.
- **A `null` value expects the argument to be absent or `null`.** The check cannot tell a
  `null` argument from a missing one.
- **A case with no `expectedToolCalls` passes this check.** It expects nothing.

This check is not MAF's `EvalChecks.ToolCallArgsMatch`. MAF's compares a number by its
.NET type and an object argument by its text, so an expected `2` never equals a called
`2` and two objects with the same members in a different order do not match. A custom
evaluator you register still receives the case's expected calls on
`EvalItem.ExpectedToolCalls`, in the form MAF reads: text, `double` and `bool` values, and
the JSON text of an object or array.

Promoting a reference run to a case writes the names of the tools that run called, with no
arguments: a run's arguments belong to that run. Add the arguments you want checked by
editing the case. The `expectedTools` field of earlier versions, a list of names that no
check read, is gone: the cases endpoint refuses a body that still carries it, with `400`
and the name of the field that replaces it. A case saved by an earlier version keeps its
names, and they now read as entries without arguments.

## Comparing two runs

A pass rate is a poor regression signal. A suite that slides from 95% to 90% still
clears a `0.85` threshold, and nothing in the run's own summary says which five cases
stopped working. `GET /api/evals/runs/{id}/diff?baseline={runId}` aligns two runs of
the same suite case by case instead:

```bash
curl "http://localhost:5081/tracon/api/evals/runs/$SECOND/diff?baseline=$FIRST"
```

Every case lands in exactly one bucket:

| Bucket | Meaning |
|---|---|
| `Regressed` | Passed on the baseline, fails now |
| `Fixed` | Failed on the baseline, passes now |
| `StillFailing` | Failed on both |
| `Unchanged` | Passed on both |
| `Added` | Only in the run being judged |
| `Removed` | Only in the baseline run |

`Added` and `Removed` are deliberately their own buckets. Adding a case to a suite
moves the pass rate without anything having broken, and a gate that cannot tell those
apart cries wolf every time someone extends a suite. Each entry names both sides'
agent run, so a regression is one click from the two conversations that produced it.

Two answers are refusals rather than results:

- **`409`** — one of the runs holds results for fewer cases than its summary counts,
  because they aged out of the `eval_case_results` retention window. Retention deletes
  in batches and can leave a run partly trimmed, so the check counts rather than merely
  looking for emptiness: comparing the survivors would report every deleted case as
  `Removed` and quietly narrow the regression count to the rows that happen to remain.
- **`400`** — the two runs measure different suites, or one of them never completed.

Cases are aligned by identifier, and a case's content is not snapshotted per run. So
keeping a case's `id` while rewriting its question compares the new question's result
against the old question's — the diff reads them as the same case because you said they
were. That is what you want for a reworded or corrected case, and not what you want for
a different question: give a different question its own case by omitting the `id`.

### As a CI gate

`tracon eval` turns the same comparison into an exit code:

```bash
tracon eval --url http://localhost:5081/tracon --suite support \
  --baseline previous --max-regressions 0
```

`--baseline` takes an eval run id or the word `previous`, which means the newest
completed run of that suite before this one. `--max-regressions` says how many cases
may break.

Three rules keep the gate honest:

- `--max-regressions` without `--baseline` is an **argument error** (exit `1`), never a
  silent no-op. A pipeline must not read a green exit code as "no regressions" when
  nothing was compared.
- A comparison that cannot be made exits **`4`**, not `3`. A lost history needs a
  different fix than a broken case.
- On a suite's first run there is nothing to compare against. That is written to
  stderr and the gate is **skipped**, not failed — otherwise every new suite's first
  CI run goes red for no reason.

The absolute thresholds still apply: `--min-pass-rate` and `--max-failures` are checked
first, and the relative gate only ever adds a check.

The eval run screen carries the same comparison, with a baseline picker and the
`Regressed` group open by default.

Cases can also be **promoted from a real run** — a production conversation that went
wrong becomes a regression case in one request:

```bash
curl -X POST http://localhost:5081/tracon/api/evals/support/cases/from-run/$RUN_ID
```

Promotion copies the run's recorded input, so the case's run sends the agent the
messages the production run sent: the request's whole text becomes `query`, the
documents it carried become `context`, and the parameter values it was bound with become
`parameters`. Failed and sessionless runs can be promoted too. Promoting the same run
twice returns the existing case rather than duplicating it.

A promotion refuses rather than build a case that leaves part of the run's input
behind:

| Answer | When |
|---|---|
| `409` | The run's session has an earlier turn — a single query cannot represent the conversation that produced the answer |
| `422` | The request carried an attachment or other non-text content, or the recorded input holds a message besides the documents and the request |
| `422` | The input was not recorded and the run carried documents |
| `422` | No request text can be read, for example a run that sent only documents |

When no input was recorded — `RunRecording:RecordRunInput` was off, or retention
removed the record — the query comes from the run's `RunStarted` event instead, and
that is the query only. Its text is cut at `RunRecording:MaxPayloadLength` (8,192
characters by default); a cut query keeps the `…[truncated]` marker, so the case shows
it is incomplete and you can correct it in the case list. The promotion's audit entry
(`eval.case.promoted`) records which source was used and whether the query was cut.

## Online evaluation

Register an `IRunJudge` and finished runs are sampled and scored automatically. The
summary endpoint reports the average score, the sample count, and what the judging
cost.

That summary is **in-memory** and resets when the process restarts. For a number
that survives a restart, use the [persistent score summary](#persistent-score-summary)
below instead. Judging costs model calls, which is why it samples rather than
scoring everything.

`POST /api/runs/{runId}/judge` scores one run immediately, skipping the sampling
decision — for calibration and debugging.

The built-in judge is configured with `ModelRunJudgeOptions`: `Criteria` states the
standard to score against, and `Instructions` replaces the judge prompt when the
default wording does not fit your domain.

For lifecycle, concurrency, timeout, tenant, and retry requirements of a custom
judge, see [Write your own judge](/guides/write-your-own-judge/).

### Calibrated evaluators

A judge does not have to be a prompt you wrote. `AddEvaluatorJudge` binds any
`IEvaluator` from `Microsoft.Extensions.AI.Evaluation` — including the calibrated
catalog in `Microsoft.Extensions.AI.Evaluation.Quality`, which grades relevance,
coherence, completeness, task adherence, and tool-call accuracy without you
writing a scoring prompt at all.

Tracon does **not** reference that catalog. Add the package when you want it,
so a consumer who does not carries none of it:

```bash
dotnet add package Microsoft.Extensions.AI.Evaluation.Quality
```

```csharp
builder.AddTracon()
       .AddEvaluatorJudge(
           "relevance",
           new RelevanceEvaluator(),
           options => options.Model = new ModelBinding { Provider = "openai", Model = "gpt-5.4-mini" });
```

Every metric the evaluator reports becomes its **own** score row, named
`{judge}.{metric}` — `relevance.Relevance`, `relevance.Coherence`. The judge-name
prefix is what lets two evaluators report a metric of the same name without one
overwriting the other, because a score name is part of what makes a score unique.

A metric the evaluator could not measure is stored with a **null** value, not a
zero, and its diagnostics are kept in the comment so the gap is explained. Metrics
map onto score kinds by shape: a numeric metric to `Numeric`, a boolean metric to
`Binary`, a string metric to `Categorical`.

Two limits are worth knowing before you turn this on:

- **Each evaluator costs a model call per sampled run.** Registering four of them
  multiplies the judging cost of online evaluation by four. Each judgment writes one
  `judge:{name}` run of kind `Eval` that carries its calls' tokens and cost, as the
  built-in judge does, and the cost reaches the judge cost metric. An evaluator that
  throws closes that run as `Failed`.
- **A bridged metric never enters the 0-100 online average.** The calibrated
  evaluators grade on their own scales, and mixing a 1-5 score into a 0-100 average
  would fire the low-score alarm on healthy runs. Only a judge's *headline* score —
  the one named exactly after the judge, as the built-in judge writes — feeds that
  window. Bridged scores are stored, returned, and aggregated by the
  [persistent score summary](#persistent-score-summary), which groups by name and
  kind and so keeps the scales apart.

A judge sees the run's input, its output text, and the **names** of the tools it
called — not their arguments or results. An evaluator that grades tool calls has
nothing to grade there and reports no measurement.

#### Which version graded this run

A bridged score is produced by that package's **prompt**, so upgrading the
evaluator package can move your scores while the model under test has not changed
at all. Read as a trend, that looks exactly like a regression.

Every score a bridged judge writes therefore carries the version of the package
that produced it:

```json
{
  "name": "relevance.Relevance",
  "source": "judge:relevance",
  "value": 78,
  "evaluatorVersion": "10.9.0+b10f9c0a081b5dbb7755b8f5592e1d3c3f550a3a"
}
```

The version is read from the evaluator's own assembly, so an evaluator you wrote
yourself is stamped on the same terms as a catalog one, and it is resolved once
when the judge is built rather than on every run. It is the informational
version, which usually carries a build revision after a `+`, so compare two
stamps for equality rather than parsing them.

`evaluatorVersion` is **null** when no version applies or none could be read — a
human score, an API score, the built-in judge (whose verdict depends on the model
you configured, not on a package), or an evaluator whose assembly carries no
version. Null means *no version was resolved*; it never means version zero, and a
judge whose version cannot be read is still scored normally. Comparing two runs
whose `evaluatorVersion` differs tells you the judge changed, not the model.

To report a version from a judge of your own, set `RunJudgment.EvaluatorVersion`;
it is written onto every score in that judgment and is bounded by
`RunScoreRules.MaxEvaluatorVersionLength`.

### Replacing what grades a suite

Eval suites are graded by MAF's `LocalEvaluator`, built from the checks the suite
declares. `AddEvalEvaluatorFactory` replaces that choice. The factory receives the
suite's compiled checks, so it can wrap the built-in behaviour rather than
discarding what the suite asked for. Registering none changes nothing.

## Human feedback

Scores can be attached to a run, or to a single message in it. Human scores and judge
scores live in **one** list with a source field on each entry, not in separate
endpoints, so "what do we think of this run" is one question.

Every score carries a **name** — `helpfulness`, `accuracy`, `severity` — and the name
is part of what makes a score unique. One reviewer can therefore score the same run
several times over, once per name, and writing the same name again updates that row
instead of opening another. A request that sends no name gets `overall`, so a client
that never asks for names keeps a single score per reviewer.

A name is a low-cardinality label matching `[A-Za-z0-9._-]{1,64}` — the same rule a
judge name follows, because a judge writes its own name onto the score it produces.
It is used as a metric tag, so a run id or a timestamp does not belong there.

A score carries one of four shapes:

| Kind | Carries | Example |
|---|---|---|
| `Binary` | `value` 0 or 1 | thumbs down / thumbs up |
| `Stars` | `value` 1 to 5 | a star rating |
| `Numeric` | `value` 0 to 100 | a judge's score, or a similarity of `0.87` |
| `Categorical` | `textValue` | `minor`, `major`, `blocking` |

Alongside the value, a score carries `evaluatorVersion`: the version of the
component that produced it, or **null** when none applies. See
[which version graded this run](#which-version-graded-this-run).

`value` is a decimal, so `0.87` is stored as `0.87`. A **null** `value` means no
measurement was made — not zero. Zero is a measurement; the absence of one is not,
which is the same rule a judge follows: a judge that cannot decide returns an empty
`RunJudgment` and writes no row at all.

Deleting a score is written to the audit trail: removing a judgement is itself
traceable.

## Persistent score summary

`GET /api/evaluation/scores/summary` aggregates the scores already written — human
and judge scores together, since they live in the same list. Unlike the
[online evaluation](#online-evaluation) summary, this one reads persisted rows, so
the same request made before and after a restart returns the same result.

Every breakdown groups by **name and kind together**: a 1-5 star rating and a 0-100
numeric score sharing a name never average into one number. `byName` is always
returned; `byAuthor`, `bySource`, and `byAgent` narrow the same data by a different
dimension. A `Categorical` group reports a count per category instead of an average.

```
GET /api/evaluation/scores/summary?bucket=day&from=2026-09-01T00:00:00Z
```

```json
{
  "byName": [
    { "key": "overall", "kind": "Binary", "count": 42, "noValueCount": 0, "average": 0.83 }
  ],
  "series": [
    { "bucketStart": "2026-09-01T00:00:00Z", "groups": [ /* one entry per (name, kind) scored that day */ ] }
  ]
}
```

`bucket` (`hour`, `day`, or `week`, UTC) adds the `series` field — a trend over time,
one entry per bucket that actually has a score. A bucket with nothing scored in it is
left out; the series is sparse, not filled. Leaving `bucket` out costs nothing extra
and returns an empty `series`. A `bucket` given with no `from` defaults the whole query
(breakdowns included) to the last 90 days — a series has no other bound the way a
breakdown does; pass `from` explicitly for an unbounded breakdown alongside it.

`messageId` is never a breakdown dimension — its cardinality is unbounded. Use
`target` (`run`, `message`, or `any`, the default) to narrow to run-level or
message-level scores instead.

Every breakdown, and the categories inside one `Categorical` group, is capped at
`maxRows` (default 20, max 500). A group's `truncatedCategoryCount` reports how many
categories did not fit, so a long tail never silently disappears.

## Experiments

An experiment splits traffic between **two versions of the same agent**. Since
code-defined agents have no version history, they cannot be experimented on.

```mermaid
flowchart LR
    accTitle: Experiment version assignment
    accDescr: An eligible agent request is assigned to the current or candidate version by a stable hash, then records that assignment on the run.
    REQ["POST /api/agents/support/run"] --> ASSIGN{"a Running experiment<br/>for this agent?"}
    ASSIGN -->|no| CUR["current version"]
    ASSIGN -->|yes| SPLIT["assign an arm by weight"]
    SPLIT --> VA["version A"]
    SPLIT --> VB["version B"]
    VA --> REC["recorded with its arm"]
    VB --> REC
```

Variant weights must sum to 100, and only one experiment per agent can be `Running` at
a time. Assignment happens **only** on `POST /api/agents/{name}/run`, synchronous or
queued (`Prefer: respond-async`). The OpenAI-compatible endpoints, inbound triggers
and child-agent calls do not go through it, which keeps the comparison to traffic
you meant to split.

The assignment key is the request's `sessionId`, or the run id when there is no
session. A session therefore lands in the same arm on both paths. A queued run gets
its arm when Tracon accepts it, and the worker runs that arm's version. The run counts
in the arm only when the worker starts it: a queued run that never starts adds
nothing to the results.

A run that carries on an earlier run keeps that run's version and arm. This applies
to the continuation after an approval decision and to the automatic continuation
after a crash. A replay is not part of the experiment.

The results endpoint gives per-arm counts, error rates, tokens, and durations. It
makes **no statistical claim about a winner**; it shows the raw numbers and leaves the
judgement to you.

Stopping affects new runs only. A run already in flight keeps its arm, and so does a
queued run that Tracon accepted before the stop. Results stay readable, and the agent
becomes free for another experiment. Deleting a `Running`
experiment is refused — stop it first, so traffic is never split against a definition
that no longer exists.

### Canary rules

A two-arm experiment can carry a canary rule: one arm is the canary and the other is
the control. The evaluation is **not persisted** — it is recomputed from current run
results on every read, so it never reports a stale verdict.

## Read next

- [Agents and definitions](/concepts/agents/) — versions, which experiments need
- [Governance](/concepts/governance/) — configure approvals, quotas, guards, and audit behavior.