Evals

15 operations. {prefix} is the route prefix passed to MapTracon; the template uses /tracon.

GET {prefix}/api/evals

Operation ID: TraconListEvalSuites

Lists a tenant’s eval suites.

Each entry is a suite’s definition — the agent under test and its check definitions — without the cases or the past runs; read those from the cases and runs endpoints. The response is not paged.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Response	Body	Headers
200 OK	application/json → array of EvalSuite	—

GET {prefix}/api/evals/{name}

Operation ID: TraconGetEvalSuite

Gets a single eval suite.

The suite carries its checks, which are stored together with it rather than as separate rows, because a suite’s checks are always read and written as one unit. Cases and runs are separate endpoints. Suites are scoped to the calling tenant, and an unknown name returns 404.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—

Response	Body	Headers
200 OK	application/json → EvalSuite	—

PUT {prefix}/api/evals/{name}

Operation ID: TraconSaveEvalSuite

Creates or updates an eval suite.

Check definitions are declarative; an unknown check type turns into an error at run time.

Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—

Request body (required):

• application/json → EvalSuiteSaveRequest

Response	Body	Headers
200 OK	application/json → EvalSuite	—

DELETE {prefix}/api/evals/{name}

Operation ID: TraconDeleteEvalSuite

Deletes an eval suite (together with its cases and runs).

The cases and every past eval run cascade with the suite, so the score history used to compare agent versions disappears with it — export it first if it matters. The agent runs those evals produced stay in the run history and are still readable there. An unknown name returns 404.

Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—

Response	Body	Headers
204 No Content	—	—

GET {prefix}/api/evals/{name}/cases

Operation ID: TraconListEvalCases

Lists a suite’s cases.

Cases come back in their stored order. Each carries an ‘id’ that is its identity for life — send it back on a replace to keep the case, and the run-to-run diff behind ‘–baseline’ will read it as the same case. The sequence number is position, not identity: reordering the list re-numbers the cases without changing which case is which. An unknown suite name returns 404, while a suite with no cases returns an empty list.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—

Response	Body	Headers
200 OK	application/json → array of EvalCase	—

PUT {prefix}/api/evals/{name}/cases

Operation ID: TraconSaveEvalCases

Replaces all of a suite’s cases with the given list.

This is a full replacement, not an append: cases missing from the body are removed, so send the complete list every time. Send each kept case back with the ‘id’ it was listed with — a case holds that id for its whole life and the run-to-run diff behind ‘–baseline’ is matched on it, so a case that arrives without one is a NEW case and an unchanged case sent without its id reads as one removed and another added. An id that does not belong to this suite, or that appears twice, fails the whole request with 400. ‘parameters’ travels the same way: it is part of the case, and a parameterized agent’s case sent without it then fails the missing-parameter check at run time. ‘expectedToolCalls’ lists the tool calls the ‘toolCallArgsMatch’ check looks for, each a ‘name’ with optional ‘arguments’; the old ‘expectedTools’ key is refused with 400. Promotion data is the server’s own and follows the id: keep the case, keep its origin. Sequence numbers are assigned from the body’s order, which means reordering the list re-numbers the cases. Every case needs a non-empty ‘query’; one that does not fails the whole request with 400 and nothing is written. An unknown suite name returns 404.

Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—

Request body (required):

• application/json → array of EvalCaseInput

Response	Body	Headers
200 OK	application/json → array of EvalCase	—

DELETE {prefix}/api/evals/{name}/cases

Operation ID: TraconClearEvalCases

Deletes all of a suite’s cases.

The suite itself survives with its checks intact; only the cases go. Past eval runs and their per-case results are kept, but they then point at cases that no longer exist. The call is idempotent — clearing an already empty suite still answers 204. An unknown suite name returns 404.

Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—

Response	Body	Headers
204 No Content	—	—

POST {prefix}/api/evals/{name}/cases/from-run/{runId}

Operation ID: TraconPromoteRunToEvalCase

Promotes a run to an eval case in a single request.

The case copies the run’s recorded input: the query, the documents it carried and the parameter values it was bound with. Without a recorded input the query comes from the RunStarted event. A run whose input a case cannot carry is refused (422). If the same run is promoted a second time, the existing case is returned (200, not 201).

Authorization: bearer authentication; Operator role policy when that policy is registered; EvalsAdmin API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—
runId	path	yes	string (uuid)	—

Request body (optional):

• application/json → EvalCasePromotionRequest

Response	Body	Headers
200 OK	application/json → EvalCase	—
201 Created	application/json → EvalCase	—

POST {prefix}/api/evals/{name}/run

Operation ID: TraconTriggerEvalRun

Runs an eval suite now.

Each case runs in a new session on the agent being evaluated and produces its own ‘runs’ row. The run is queued as a job; results are processed in the background.

Authorization: bearer authentication; Operator role policy when that policy is registered; RunsWrite API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—

Request body (optional):

• application/json → EvalRunTriggerRequest

Response	Body	Headers
200 OK	application/json → EvalRun	—

GET {prefix}/api/evals/{name}/runs

Operation ID: TraconListEvalRuns

Lists a suite’s past runs.

Each entry is one execution of the whole suite with its aggregate outcome; the per-case results live behind the single eval-run endpoint. To find the regression between two of these entries, hand both to the eval-run diff endpoint: it aligns them case by case instead of leaving the comparison to the caller. Paging is offset based, with ‘skip’ defaulting to 0 and ‘take’ to 50. An unknown suite name returns 404.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Parameter	In	Required	Type	Description and rules
name	path	yes	string	—
skip	query	no	integer (int32)	pattern `^-?(?:0\
take	query	no	integer (int32)	pattern `^-?(?:0\

Response	Body	Headers
200 OK	application/json → array of EvalRun	—

GET {prefix}/api/evals/runs/{id}

Operation ID: TraconGetEvalRun

Gets a single eval run and its per-case results.

This is the endpoint to poll after triggering a suite: the eval run is queued and processed in the background, and its results fill in as cases complete. Each result names the agent run it came from, so a failing check can be traced to the exact conversation. Per-case results are a retention target, so an old eval run may keep its summary while its details are gone. An unknown id, or one belonging to another tenant, returns 404.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Parameter	In	Required	Type	Description and rules
id	path	yes	string (uuid)	—

Response	Body	Headers
200 OK	application/json → EvalRunDetailResponse	—

GET {prefix}/api/evals/runs/{id}/diff

Operation ID: TraconDiffEvalRuns

Compares two eval runs of the same suite, case by case.

The run in the path is the candidate; ‘baseline’ names the run it is judged against. Every case lands in exactly one bucket - Regressed, Fixed, StillFailing, Unchanged, Added or Removed - and each entry names both sides’ agent run, so a regression is one click from the two conversations that produced it. Cases added to or dropped from the suite are their own buckets and are never counted as regressions. Paging is offset based over the aligned cases, regressions first; the counters always describe the whole comparison. Both runs must have completed and must measure the same suite, otherwise 400. If retention has removed either run’s per-case results the answer is 409, never an empty diff: an empty diff would read as ‘nothing changed’. An unknown id, or one belonging to another tenant, returns 404.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Parameter	In	Required	Type	Description and rules
id	path	yes	string (uuid)	—
baseline	query	yes	string (uuid)	—
skip	query	no	integer (int32)	pattern `^-?(?:0\
take	query	no	integer (int32)	pattern `^-?(?:0\

Response	Body	Headers
200 OK	application/json → EvalRunDiff	—
400 Bad Request	application/problem+json → ProblemDetails	—
404 Not Found	application/problem+json → ProblemDetails	—
409 Conflict	application/problem+json → ProblemDetails	—

GET {prefix}/api/evaluation/online

Operation ID: TraconGetOnlineEvaluationSummary

Returns a summary of the online evaluation window.

Returns the average judge score, sample count, and judge cost within the window. The summary is in-memory (it resets when the process restarts); for an authoritative result that survives a restart, use ‘GET /api/evaluation/scores/summary’ instead.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Response	Body	Headers
200 OK	application/json → OnlineEvaluationSummary	—

GET {prefix}/api/evaluation/scores/summary

Operation ID: TraconGetRunScoreSummary

Aggregates run and message scores by name, author, source, and agent.

A query over the scores already written, not a counter – unlike ‘/api/evaluation/online’, the result survives a process restart. Each breakdown groups by (name, kind): a 1-5 star rating and a 0-100 numeric score sharing a name never average together. ‘messageId’ is never a breakdown dimension; use ‘target’ (run, message, or both) instead. ‘bucket’ (hour, day, or week, UTC) adds a trend series; omitting it costs nothing extra. A bucketed series with no ‘from’ defaults to the last 90 days, since a series has no other bound the way a breakdown does. Every breakdown, and the categories inside one categorical score’s entry, is capped at ‘maxRows’. 400 if ‘from’ is at or after ‘to’, or ‘maxRows’ is out of range.

Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.

Parameter	In	Required	Type	Description and rules
from	query	no	string (date-time)	—
to	query	no	string (date-time)	—
scoreName	query	no	string	—
agentName	query	no	string	—
source	query	no	string	—
author	query	no	string	—
target	query	no	RunScoreTarget	—
bucket	query	no	RunScoreBucket	—
maxRows	query	no	integer (int32)	pattern `^-?(?:0\

Response	Body	Headers
200 OK	application/json → RunScoreSummary	—
400 Bad Request	application/problem+json → ProblemDetails	—

POST {prefix}/api/runs/{runId}/judge

Operation ID: TraconJudgeRun

Manually has judge(s) score a run.

This SKIPS the sampling decision; it is for calibration and debugging. If no IRunJudge is registered, or the run’s input/output cannot be read, an empty list is returned. Judging both READS the run and WRITES a score for it, so a registered IRunAuthorizationHandler is asked for both; either denial returns 404, identical to a run that does not exist.

Authorization: bearer authentication; Operator role policy when that policy is registered; RunsWrite API-key scope.

Parameter	In	Required	Type	Description and rules
runId	path	yes	string (uuid)	—

Response	Body	Headers
200 OK	application/json → JudgeRunResponse	—