IEvalEvaluatorFactory

Namespace Tracon · Assembly Tracon.Core.dll

The extension point that builds the AI.IAgentEvaluator an eval suite is graded with.

public interface IEvalEvaluatorFactory

Remarks

Tracon grades a suite with MAF’s AI.LocalEvaluator, built from the checks the suite declares. This interface is the seam around that choice: a consumer registers its own factory to grade with a different evaluator, or to wrap the built-in one.

The factory receives the suite’s compiled checks rather than being a bare AI.IAgentEvaluator singleton. A suite declares its checks and Tracon refuses to run a suite that declares none; an evaluator that never saw them would silently grade something else. A factory that ignores the argument takes over grading completely, and that is then a deliberate choice rather than an accident.

The factory is a singleton and is called once per eval run. It must be thread-safe. Registering none keeps today’s behaviour exactly.

Tenant mode: tenant-independent. The factory sees only the suite’s checks, never a tenant identifier or tenant data; the run it grades carries the tenant, and the eval store applies it.

Delivery: no delivery guarantee applies. This is a synchronous factory call on the eval job’s own thread, not a dispatch mechanism. It is called again when a job’s lease expires and the job is retried, so it must be cheap and free of side effects.

Methods

Create(IReadOnlyList<EvalCheck>)

Creates the evaluator that grades one eval run.

IAgentEvaluator Create(IReadOnlyList<EvalCheck> checks)

Parameters

checks IReadOnlyList<EvalCheck>

The suite’s compiled checks. Never empty.

Returns

IAgentEvaluator

The evaluator. Must not be null.