IEvalEvaluatorFactory

Namespace Tracon · Assembly Tracon.Core.dll

The extension point that builds the AI.IAgentEvaluator an eval suite is graded with.

public interface IEvalEvaluatorFactory

Remarks

Tracon grades a suite with MAF’s AI.LocalEvaluator, built from the checks the suite declares. This interface is the seam around that choice: a consumer registers its own factory to grade with a different evaluator, or to wrap the built-in one.

The factory receives the suite’s compiled checks rather than being a bare AI.IAgentEvaluator singleton. A suite declares its checks and Tracon refuses to run a suite that declares none; an evaluator that never saw them would silently grade something else. A factory that ignores the argument takes over grading completely, and that is then a deliberate choice rather than an accident.

The factory is a singleton and is called once per eval run. It must be thread-safe. Registering none keeps today’s behaviour exactly.

The built-in toolCallArgsMatch check reads a case’s expected tool calls from the very AI.EvalItem instance the run builds. The item’s ExpectedToolCalls carries them too, in the form Microsoft Agent Framework reads, where an object or array value is its JSON text. An evaluator that copies the items before it hands them to the checks leaves that check with the second form only: an expected object then compares as text and does not equal the object a call carried. Pass the items through unchanged, or compare in a check of your own.

Tenant mode: tenant-independent. The factory sees only the suite’s checks, never a tenant identifier or tenant data; the run it grades carries the tenant, and the eval store applies it.

Delivery: no delivery guarantee applies. This is a synchronous factory call on the eval job’s own thread, not a dispatch mechanism. It is called again when a job’s lease expires and the job is retried, so it must be cheap and free of side effects.

Methods

Create(IReadOnlyList<EvalCheck>)

Creates the evaluator that grades one eval run.

IAgentEvaluator Create(IReadOnlyList<EvalCheck> checks)

Parameters

checks IReadOnlyList<EvalCheck>

The suite’s compiled checks. Never empty.

Returns

IAgentEvaluator

The evaluator. Must not be null.