Test and evaluate

Evaluation and experiments

Use evaluation suites, judges, experiments, and regression checks to compare agent behavior against explicit criteria. They answer different questions and are meant to be used together.

	Question	When it runs
Eval suites	Did this change break anything?	On demand, against a fixed case set
Online judges	Is production drifting?	Continuously, on sampled live runs
Human feedback	What do people think?	Whenever someone scores a run
Experiments	Is version B better than A?	Live, splitting real traffic

Eval suites

A suite names the agent under test and carries declarative checks — a JSON array stored and read as one unit with the suite. Create or update the suite before adding cases:

curl -X PUT http://localhost:5081/tracon/api/evals/support \
     -H 'Content-Type: application/json' \
     -d '{
       "agentName": "support",
       "checks": [{"kind":"toolCalled","tools":["get_order_status"]}]
     }'

A suite needs at least one check before it can run — an empty checks array fails the run outright instead of reporting every case as passed. Seven built-in kinds cover the common cases; six match Microsoft.Agents.AI.EvalChecks factories directly and toolCallArgsMatch is Tracon’s own:

Kind	Checks	Fields
`nonEmpty`	The response has at least `minLength` characters	`minLength` (default 1)
`containsExpected`	The response contains the case’s `expectedOutput`	`caseSensitive` (default false)
`keywords`	The response contains every string in `values`	`values`, `caseSensitive`
`toolCalled`	The listed `tools` were called, `all` or `any` of them	`tools`, `mode` (default `all`)
`toolCallsPresent`	At least one tool was called	—
`toolCallArgsMatch`	Every tool call in the case’s `expectedToolCalls` was made, with the listed arguments	— (reads the case)
`hasImageContent`	The response carries image content	—

Application code can add more with ITraconBuilder.AddEvalCheck(kind, check) — a named MAF EvalCheck that becomes usable under a custom kind name alongside the seven built-in ones. A kind that matches neither fails the run with a clear error instead of being silently skipped.

Cases are a separate, ordered list of queries:

curl -X PUT http://localhost:5081/tracon/api/evals/support/cases \
     -H 'Content-Type: application/json' \
     -d '[{"id":"01a0b449-ac16-712d-8bd8-1a60155d0d87",
          "query":"Where is order 4182?","expectedOutput":"shipped"},
         {"query":"Has order 4182 shipped yet?","expectedOutput":"shipped"}]'

That PUT is a full replacement: cases missing from the body are removed, so send the whole list every time. Send each case you are keeping back with the id it was listed with — a case holds that identifier for life, and runs are compared by it (see below). A case that arrives without an id is a new case, so an unchanged case sent without its id reads as one removed and another added; that is also how a real regression in the same window escapes --max-regressions, which ignores removals by design. An id that does not belong to this suite, or that appears twice, fails the whole request with 400 and nothing is written. parameters travels back the same way: it is part of the case, not of the request. Sequence numbers come from the body’s order, so reordering re-numbers the cases — but a sequence number is only display order and does not change which case is which. expectedOutput reaches containsExpected. The tool names a toolCalled check verifies come from the suite’s own check definition, shown above; the calls a toolCallArgsMatch check verifies come from the case itself.

A case’s context is reference text for the agent. It reaches the agent through the document channel, ahead of the query — the messages a run with documents sends. Plain text goes as one document named context. Text made only of document blocks (-----BEGIN TRACON DOCUMENT----- … -----END TRACON DOCUMENT-----), which is what a promoted run’s documents become, goes back as one document per block, unchanged. The evaluator receives the same text as the case’s context, for checks that judge an answer against reference material.

For an agent that declares parameters, a case’s own parameters field supplies the values that run’s instructions bind against. A case missing a value the agent requires fails outright, with a reason naming which parameter is missing — the same check POST /api/agents/{name}/run applies, so a case that would fail in production fails here too, before any model call is made. The case run records the values it was bound with, defaults included, as every other run does.

Running a suite queues a job. Each case runs in its own fresh session against the agent and produces its own run row, so a failing check can be traced to the exact conversation that produced it. That session is not stored, so a case whose run asks for a tool approval fails with a reason that says so, and its run is closed as Failed: nobody can answer the approval.

curl -X POST http://localhost:5081/tracon/api/evals/support/run
curl http://localhost:5081/tracon/api/evals/support/runs

Checking which tool was called with which arguments

A case lists the tool calls it expects in expectedToolCalls. Each entry is a tool name and, optionally, the arguments that call must carry:

curl -X PUT http://localhost:5081/tracon/api/evals/support \
     -H 'Content-Type: application/json' \
     -d '{"agentName":"support","checks":[{"kind":"toolCallArgsMatch"}]}'

curl -X PUT http://localhost:5081/tracon/api/evals/support/cases \
     -H 'Content-Type: application/json' \
     -d '[{"query":"Where is order ORD-1001?",
          "expectedToolCalls":[
            {"name":"get_order_status","arguments":{"orderId":"ORD-1001"}},
            {"name":"get_shipping_status"}]}]'

The case passes the check when, for every entry, the agent called a tool of that name (letter case does not matter) and, when the entry lists arguments, one such call carried every listed argument with an equal value. An entry without arguments checks only that the tool was called. A failing result names each entry and what was wrong — not called or args mismatch — without repeating the argument values the model sent.

The listed arguments are a subset. The call may carry others, and an argument name matches in any letter case.
A value compares by its JSON meaning. 2 equals 2.0, an object equals an object with the same properties in any order, and spacing between tokens does not matter. A nested object or array must match exactly; the subset rule applies to the top-level argument names only.
A null value expects the argument to be absent or null. The check cannot tell a null argument from a missing one.
A case with no expectedToolCalls passes this check. It expects nothing.

This check is not MAF’s EvalChecks.ToolCallArgsMatch. MAF’s compares a number by its .NET type and an object argument by its text, so an expected 2 never equals a called 2 and two objects with the same members in a different order do not match. A custom evaluator you register still receives the case’s expected calls on EvalItem.ExpectedToolCalls, in the form MAF reads: text, double and bool values, and the JSON text of an object or array.

Promoting a reference run to a case writes the names of the tools that run called, with no arguments: a run’s arguments belong to that run. Add the arguments you want checked by editing the case. The expectedTools field of earlier versions, a list of names that no check read, is gone: the cases endpoint refuses a body that still carries it, with 400 and the name of the field that replaces it. A case saved by an earlier version keeps its names, and they now read as entries without arguments.

Comparing two runs

A pass rate is a poor regression signal. A suite that slides from 95% to 90% still clears a 0.85 threshold, and nothing in the run’s own summary says which five cases stopped working. GET /api/evals/runs/{id}/diff?baseline={runId} aligns two runs of the same suite case by case instead:

curl "http://localhost:5081/tracon/api/evals/runs/$SECOND/diff?baseline=$FIRST"

Every case lands in exactly one bucket:

Bucket	Meaning
`Regressed`	Passed on the baseline, fails now
`Fixed`	Failed on the baseline, passes now
`StillFailing`	Failed on both
`Unchanged`	Passed on both
`Added`	Only in the run being judged
`Removed`	Only in the baseline run

Added and Removed are deliberately their own buckets. Adding a case to a suite moves the pass rate without anything having broken, and a gate that cannot tell those apart cries wolf every time someone extends a suite. Each entry names both sides’ agent run, so a regression is one click from the two conversations that produced it.

Two answers are refusals rather than results:

409 — one of the runs holds results for fewer cases than its summary counts, because they aged out of the eval_case_results retention window. Retention deletes in batches and can leave a run partly trimmed, so the check counts rather than merely looking for emptiness: comparing the survivors would report every deleted case as Removed and quietly narrow the regression count to the rows that happen to remain.
400 — the two runs measure different suites, or one of them never completed.

Cases are aligned by identifier, and a case’s content is not snapshotted per run. So keeping a case’s id while rewriting its question compares the new question’s result against the old question’s — the diff reads them as the same case because you said they were. That is what you want for a reworded or corrected case, and not what you want for a different question: give a different question its own case by omitting the id.

As a CI gate

tracon eval turns the same comparison into an exit code:

tracon eval --url http://localhost:5081/tracon --suite support \
  --baseline previous --max-regressions 0

--baseline takes an eval run id or the word previous, which means the newest completed run of that suite before this one. --max-regressions says how many cases may break.

Three rules keep the gate honest:

--max-regressions without --baseline is an argument error (exit 1), never a silent no-op. A pipeline must not read a green exit code as “no regressions” when nothing was compared.
A comparison that cannot be made exits 4, not 3. A lost history needs a different fix than a broken case.
On a suite’s first run there is nothing to compare against. That is written to stderr and the gate is skipped, not failed — otherwise every new suite’s first CI run goes red for no reason.

The absolute thresholds still apply: --min-pass-rate and --max-failures are checked first, and the relative gate only ever adds a check.

The eval run screen carries the same comparison, with a baseline picker and the Regressed group open by default.

Cases can also be promoted from a real run — a production conversation that went wrong becomes a regression case in one request:

curl -X POST http://localhost:5081/tracon/api/evals/support/cases/from-run/$RUN_ID

Promotion copies the run’s recorded input, so the case’s run sends the agent the messages the production run sent: the request’s whole text becomes query, the documents it carried become context, and the parameter values it was bound with become parameters. Failed and sessionless runs can be promoted too. Promoting the same run twice returns the existing case rather than duplicating it.

A promotion refuses rather than build a case that leaves part of the run’s input behind:

Answer	When
`409`	The run’s session has an earlier turn — a single query cannot represent the conversation that produced the answer
`422`	The request carried an attachment or other non-text content, or the recorded input holds a message besides the documents and the request
`422`	The input was not recorded and the run carried documents
`422`	No request text can be read, for example a run that sent only documents

When no input was recorded — RunRecording:RecordRunInput was off, or retention removed the record — the query comes from the run’s RunStarted event instead, and that is the query only. Its text is cut at RunRecording:MaxPayloadLength (8,192 characters by default); a cut query keeps the …[truncated] marker, so the case shows it is incomplete and you can correct it in the case list. The promotion’s audit entry (eval.case.promoted) records which source was used and whether the query was cut.

Online evaluation

Register an IRunJudge and finished runs are sampled and scored automatically. The summary endpoint reports the average score, the sample count, and what the judging cost.

That summary is in-memory and resets when the process restarts. For a number that survives a restart, use the persistent score summary below instead. Judging costs model calls, which is why it samples rather than scoring everything.

POST /api/runs/{runId}/judge scores one run immediately, skipping the sampling decision — for calibration and debugging.

The built-in judge is configured with ModelRunJudgeOptions: Criteria states the standard to score against, and Instructions replaces the judge prompt when the default wording does not fit your domain.

For lifecycle, concurrency, timeout, tenant, and retry requirements of a custom judge, see Write your own judge.

Calibrated evaluators

A judge does not have to be a prompt you wrote. AddEvaluatorJudge binds any IEvaluator from Microsoft.Extensions.AI.Evaluation — including the calibrated catalog in Microsoft.Extensions.AI.Evaluation.Quality, which grades relevance, coherence, completeness, task adherence, and tool-call accuracy without you writing a scoring prompt at all.

Tracon does not reference that catalog. Add the package when you want it, so a consumer who does not carries none of it:

dotnet add package Microsoft.Extensions.AI.Evaluation.Quality

builder.AddTracon()
       .AddEvaluatorJudge(
           "relevance",
           new RelevanceEvaluator(),
           options => options.Model = new ModelBinding { Provider = "openai", Model = "gpt-5.4-mini" });

Every metric the evaluator reports becomes its own score row, named {judge}.{metric} — relevance.Relevance, relevance.Coherence. The judge-name prefix is what lets two evaluators report a metric of the same name without one overwriting the other, because a score name is part of what makes a score unique.

A metric the evaluator could not measure is stored with a null value, not a zero, and its diagnostics are kept in the comment so the gap is explained. Metrics map onto score kinds by shape: a numeric metric to Numeric, a boolean metric to Binary, a string metric to Categorical.

Two limits are worth knowing before you turn this on:

Each evaluator costs a model call per sampled run. Registering four of them multiplies the judging cost of online evaluation by four. Each judgment writes one judge:{name} run of kind Eval that carries its calls’ tokens and cost, as the built-in judge does, and the cost reaches the judge cost metric. An evaluator that throws closes that run as Failed.
A bridged metric never enters the 0-100 online average. The calibrated evaluators grade on their own scales, and mixing a 1-5 score into a 0-100 average would fire the low-score alarm on healthy runs. Only a judge’s headline score — the one named exactly after the judge, as the built-in judge writes — feeds that window. Bridged scores are stored, returned, and aggregated by the persistent score summary, which groups by name and kind and so keeps the scales apart.

A judge sees the run’s input, its output text, and the names of the tools it called — not their arguments or results. An evaluator that grades tool calls has nothing to grade there and reports no measurement.

Which version graded this run

A bridged score is produced by that package’s prompt, so upgrading the evaluator package can move your scores while the model under test has not changed at all. Read as a trend, that looks exactly like a regression.

Every score a bridged judge writes therefore carries the version of the package that produced it:

{
  "name": "relevance.Relevance",
  "source": "judge:relevance",
  "value": 78,
  "evaluatorVersion": "10.9.0+b10f9c0a081b5dbb7755b8f5592e1d3c3f550a3a"
}

The version is read from the evaluator’s own assembly, so an evaluator you wrote yourself is stamped on the same terms as a catalog one, and it is resolved once when the judge is built rather than on every run. It is the informational version, which usually carries a build revision after a +, so compare two stamps for equality rather than parsing them.

evaluatorVersion is null when no version applies or none could be read — a human score, an API score, the built-in judge (whose verdict depends on the model you configured, not on a package), or an evaluator whose assembly carries no version. Null means no version was resolved; it never means version zero, and a judge whose version cannot be read is still scored normally. Comparing two runs whose evaluatorVersion differs tells you the judge changed, not the model.

To report a version from a judge of your own, set RunJudgment.EvaluatorVersion; it is written onto every score in that judgment and is bounded by RunScoreRules.MaxEvaluatorVersionLength.

Replacing what grades a suite

Eval suites are graded by MAF’s LocalEvaluator, built from the checks the suite declares. AddEvalEvaluatorFactory replaces that choice. The factory receives the suite’s compiled checks, so it can wrap the built-in behaviour rather than discarding what the suite asked for. Registering none changes nothing.

Human feedback

Scores can be attached to a run, or to a single message in it. Human scores and judge scores live in one list with a source field on each entry, not in separate endpoints, so “what do we think of this run” is one question.

Every score carries a name — helpfulness, accuracy, severity — and the name is part of what makes a score unique. One reviewer can therefore score the same run several times over, once per name, and writing the same name again updates that row instead of opening another. A request that sends no name gets overall, so a client that never asks for names keeps a single score per reviewer.

A name is a low-cardinality label matching [A-Za-z0-9._-]{1,64} — the same rule a judge name follows, because a judge writes its own name onto the score it produces. It is used as a metric tag, so a run id or a timestamp does not belong there.

A score carries one of four shapes:

Kind	Carries	Example
`Binary`	`value` 0 or 1	thumbs down / thumbs up
`Stars`	`value` 1 to 5	a star rating
`Numeric`	`value` 0 to 100	a judge’s score, or a similarity of `0.87`
`Categorical`	`textValue`	`minor`, `major`, `blocking`

Alongside the value, a score carries evaluatorVersion: the version of the component that produced it, or null when none applies. See which version graded this run.

value is a decimal, so 0.87 is stored as 0.87. A null value means no measurement was made — not zero. Zero is a measurement; the absence of one is not, which is the same rule a judge follows: a judge that cannot decide returns an empty RunJudgment and writes no row at all.

Deleting a score is written to the audit trail: removing a judgement is itself traceable.

Persistent score summary

GET /api/evaluation/scores/summary aggregates the scores already written — human and judge scores together, since they live in the same list. Unlike the online evaluation summary, this one reads persisted rows, so the same request made before and after a restart returns the same result.

Every breakdown groups by name and kind together: a 1-5 star rating and a 0-100 numeric score sharing a name never average into one number. byName is always returned; byAuthor, bySource, and byAgent narrow the same data by a different dimension. A Categorical group reports a count per category instead of an average.

GET /api/evaluation/scores/summary?bucket=day&from=2026-09-01T00:00:00Z

{
  "byName": [
    { "key": "overall", "kind": "Binary", "count": 42, "noValueCount": 0, "average": 0.83 }
  ],
  "series": [
    { "bucketStart": "2026-09-01T00:00:00Z", "groups": [ /* one entry per (name, kind) scored that day */ ] }
  ]
}

bucket (hour, day, or week, UTC) adds the series field — a trend over time, one entry per bucket that actually has a score. A bucket with nothing scored in it is left out; the series is sparse, not filled. Leaving bucket out costs nothing extra and returns an empty series. A bucket given with no from defaults the whole query (breakdowns included) to the last 90 days — a series has no other bound the way a breakdown does; pass from explicitly for an unbounded breakdown alongside it.

messageId is never a breakdown dimension — its cardinality is unbounded. Use target (run, message, or any, the default) to narrow to run-level or message-level scores instead.

Every breakdown, and the categories inside one Categorical group, is capped at maxRows (default 20, max 500). A group’s truncatedCategoryCount reports how many categories did not fit, so a long tail never silently disappears.

Experiments

An experiment splits traffic between two versions of the same agent. Since code-defined agents have no version history, they cannot be experimented on.

flowchart LR
    accTitle: Experiment version assignment
    accDescr: An eligible agent request is assigned to the current or candidate version by a stable hash, then records that assignment on the run.
    REQ["POST /api/agents/support/run"] --> ASSIGN{"a Running experiment<br/>for this agent?"}
    ASSIGN -->|no| CUR["current version"]
    ASSIGN -->|yes| SPLIT["assign an arm by weight"]
    SPLIT --> VA["version A"]
    SPLIT --> VB["version B"]
    VA --> REC["recorded with its arm"]
    VB --> REC

Variant weights must sum to 100, and only one experiment per agent can be Running at a time. Assignment happens only on POST /api/agents/{name}/run, synchronous or queued (Prefer: respond-async). The OpenAI-compatible endpoints, inbound triggers and child-agent calls do not go through it, which keeps the comparison to traffic you meant to split.

The assignment key is the request’s sessionId, or the run id when there is no session. A session therefore lands in the same arm on both paths. A queued run gets its arm when Tracon accepts it, and the worker runs that arm’s version. The run counts in the arm only when the worker starts it: a queued run that never starts adds nothing to the results.

A run that carries on an earlier run keeps that run’s version and arm. This applies to the continuation after an approval decision and to the automatic continuation after a crash. A replay is not part of the experiment.

The results endpoint gives per-arm counts, error rates, tokens, and durations. It makes no statistical claim about a winner; it shows the raw numbers and leaves the judgement to you.

Stopping affects new runs only. A run already in flight keeps its arm, and so does a queued run that Tracon accepted before the stop. Results stay readable, and the agent becomes free for another experiment. Deleting a Running experiment is refused — stop it first, so traffic is never split against a definition that no longer exists.

Canary rules

A two-arm experiment can carry a canary rule: one arm is the canary and the other is the control. The evaluation is not persisted — it is recomputed from current run results on every read, so it never reports a stale verdict.