.NET API
IEvalStore
Tracon.Abstractions.dllThe store for evaluation (eval) suites, cases, and runs.
public interface IEvalStoreRemarks
Section titled “Remarks”DI lifetime — singleton. Registered as a singleton with
TryAdd; a consumer’s own registration wins.
Methods
Section titled “Methods”AddCaseAsync(Guid, EvalCaseDraft, CancellationToken)
Section titled “ AddCaseAsync(Guid, EvalCaseDraft, CancellationToken)”Adds a SINGLE case to the suite. Seq is generated atomically by
the store; the caller does not compute it. If the same
EvalCaseDraft.SourceRunId is added a second time, the
existing case is returned with Created: false.
ValueTask<EvalCaseAddResult> AddCaseAsync(Guid suiteId, EvalCaseDraft draft, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
draft EvalCaseDraft
The draft of the case to add.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The added (or already existing) case, and whether it was created.
CompleteRunAsync(EvalRunCompletion, CancellationToken)
Section titled “ CompleteRunAsync(EvalRunCompletion, CancellationToken)”Finalizes a run and writes its summary counters.
ValueTask CompleteRunAsync(EvalRunCompletion completion, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”completion EvalRunCompletion
The finalization information.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
CreateRunAsync(EvalRun, CancellationToken)
Section titled “ CreateRunAsync(EvalRun, CancellationToken)”Creates a new run record (state EvalRunStatus.Pending).
ValueTask<EvalRun> CreateRunAsync(EvalRun run, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”run EvalRun
The run record.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The run record with its identifier assigned.
DeleteSuiteAsync(string, string, CancellationToken)
Section titled “ DeleteSuiteAsync(string, string, CancellationToken)”Deletes the suite and all of its cases/runs (cascade).
ValueTask<bool> DeleteSuiteAsync(string tenantId, string name, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
name string
The suite name.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”true if the suite was deleted.
DiffRunsAsync(EvalRunDiffQuery, CancellationToken)
Section titled “ DiffRunsAsync(EvalRunDiffQuery, CancellationToken)”Aligns two completed runs of the same suite, case by case.
ValueTask<EvalRunDiff?> DiffRunsAsync(EvalRunDiffQuery query, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”query EvalRunDiffQuery
Which two runs to compare, and which page of the result to return.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The difference; null if either run does not exist or belongs to another tenant.
Exceptions
Section titled “Exceptions”EvalRunDiffUnavailableException
The two runs exist but cannot be compared: they measure different suites, one of them has not completed, or one of them holds results for fewer cases than its summary counts because retention removed some of them. An empty diff is never returned in their place — it would read as “nothing changed” and turn a CI gate green over a regression. Implementations delegate to EvalRunDiffBuilder.Build rather than restating the rules.
GetRunAsync(string, Guid, CancellationToken)
Section titled “ GetRunAsync(string, Guid, CancellationToken)”Fetches a single run record.
ValueTask<EvalRun?> GetRunAsync(string tenantId, Guid evalRunId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
evalRunId Guid
The run identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The record; null if it does not exist or belongs to another tenant.
GetRunByJobIdAsync(string, Guid, CancellationToken)
Section titled “ GetRunByJobIdAsync(string, Guid, CancellationToken)”Fetches the run a job record produced.
ValueTask<EvalRun?> GetRunByJobIdAsync(string tenantId, Guid jobId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
jobId Guid
The job identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The record; null if it does not exist.
GetSuiteAsync(string, string, CancellationToken)
Section titled “ GetSuiteAsync(string, string, CancellationToken)”Fetches the suite matching the given name within the tenant.
ValueTask<EvalSuite?> GetSuiteAsync(string tenantId, string name, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
name string
The suite name.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The suite; null if it does not exist.
ListCaseResultsAsync(string, Guid, CancellationToken)
Section titled “ ListCaseResultsAsync(string, Guid, CancellationToken)”Lists a run’s case results.
ValueTask<IReadOnlyList<EvalCaseResult>> ListCaseResultsAsync(string tenantId, Guid evalRunId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
evalRunId Guid
The run identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCaseResult>>
The results.
ListCasesAsync(Guid, CancellationToken)
Section titled “ ListCasesAsync(Guid, CancellationToken)”Lists a suite’s cases, ordered by sequence number.
ValueTask<IReadOnlyList<EvalCase>> ListCasesAsync(Guid suiteId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCase>>
The cases.
ListSuitesAsync(string, CancellationToken)
Section titled “ ListSuitesAsync(string, CancellationToken)”Lists all of a tenant’s suites.
ValueTask<IReadOnlyList<EvalSuite>> ListSuitesAsync(string tenantId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalSuite>>
The suites, ordered by name.
MarkRunRunningAsync(Guid, int?, string?, CancellationToken)
Section titled “ MarkRunRunningAsync(Guid, int?, string?, CancellationToken)”Transitions a run to EvalRunStatus.Running and records the agent version and model identifier being measured.
ValueTask MarkRunRunningAsync(Guid evalRunId, int? agentVersion, string? modelId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”evalRunId Guid
The run identifier.
agentVersion int?
The agent definition version being measured.
modelId string?
The model identifier being measured.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
QueryRunsAsync(EvalRunQuery, CancellationToken)
Section titled “ QueryRunsAsync(EvalRunQuery, CancellationToken)”Lists runs by filter. The newest record is returned first.
ValueTask<IReadOnlyList<EvalRun>> QueryRunsAsync(EvalRunQuery query, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”query EvalRunQuery
The filter.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalRun>>
The records.
RecordCaseResultAsync(EvalCaseResult, CancellationToken)
Section titled “ RecordCaseResultAsync(EvalCaseResult, CancellationToken)”Records a case result.
ValueTask RecordCaseResultAsync(EvalCaseResult result, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”result EvalCaseResult
The case result.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
ReplaceCasesAsync(Guid, IReadOnlyList<EvalCase>, CancellationToken)
Section titled “ ReplaceCasesAsync(Guid, IReadOnlyList<EvalCase>, CancellationToken)”Replaces all of a suite’s cases with the given list.
ValueTask<IReadOnlyList<EvalCase>> ReplaceCasesAsync(Guid suiteId, IReadOnlyList<EvalCase> cases, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
cases IReadOnlyList<EvalCase>
The new case list. Sequence numbers are assigned by list order.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCase>>
The cases with identifiers assigned.
SaveSuiteAsync(EvalSuite, CancellationToken)
Section titled “ SaveSuiteAsync(EvalSuite, CancellationToken)”Creates or updates the suite.
ValueTask<EvalSuite> SaveSuiteAsync(EvalSuite suite, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suite EvalSuite
The suite to save.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The suite with its identifier and timestamps assigned.