TraconOnlineEvaluationBuilderExtensions

Namespace Tracon · Assembly Tracon.Core.dll

Chain extensions that turn on the built-in model-based judge.

public static class TraconOnlineEvaluationBuilderExtensions

Inheritance

object ← TraconOnlineEvaluationBuilderExtensions

Inherited Members

object.GetType(), object.MemberwiseClone(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()

Remarks

This is an extension method, not a member of ITraconBuilder: adding a member to the interface is a breaking change after release, adding an extension method is not.

Methods

AddEvalEvaluatorFactory<TFactory>(ITraconBuilder)

Registers the factory that builds the AI.IAgentEvaluator an eval suite is graded with.

public static ITraconBuilder AddEvalEvaluatorFactory<TFactory>(this ITraconBuilder builder) where TFactory : class, IEvalEvaluatorFactory

Parameters

builder ITraconBuilder

Configuration chain.

Returns

ITraconBuilder

The continuation of the chain.

Type Parameters

TFactory

The factory implementation type.

Remarks

The consumer’s registration wins: the built-in factory is registered with TryAdd, so this one replaces it. Registering none leaves MAF’s AI.LocalEvaluator, built over the suite’s own checks.

builder.AddTracon()
       .AddEvalEvaluatorFactory<LoggingEvaluatorFactory>();

Exceptions

ArgumentNullException

builder is null.

AddEvalEvaluatorFactory(ITraconBuilder, IEvalEvaluatorFactory)

Registers a configured eval-evaluator factory instance.

public static ITraconBuilder AddEvalEvaluatorFactory(this ITraconBuilder builder, IEvalEvaluatorFactory factory)

Parameters

builder ITraconBuilder

Configuration chain.

factory IEvalEvaluatorFactory

The factory instance. The caller owns it.

Returns

ITraconBuilder

The continuation of the chain.

Remarks

The consumer’s registration wins over the built-in one.

builder.AddTracon()
       .AddEvalEvaluatorFactory(new LoggingEvaluatorFactory());

Exceptions

ArgumentNullException

builder or factory is null.

AddEvaluatorJudge(ITraconBuilder, string, IEvaluator)

Registers a Microsoft.Extensions.AI Evaluation.IEvaluator as a Tracon run judge.

public static ITraconBuilder AddEvaluatorJudge(this ITraconBuilder builder, string name, IEvaluator evaluator)

Parameters

builder ITraconBuilder

Configuration chain.

name string

The judge name, matching [A-Za-z0-9._-]{1,64}.

evaluator IEvaluator

The evaluator to grade sampled runs with.

Returns

ITraconBuilder

The continuation of the chain.

Remarks

The calibrated evaluator catalog lives in the Microsoft.Extensions.AI.Evaluation.Quality package, which Tracon deliberately does not reference: add it yourself when you want the catalog, and a consumer who does not use it carries no extra dependency.

Every metric the evaluator reports becomes its own score row, named {name}.{metric}. The prefix is what keeps two evaluators that report the same metric from overwriting each other, because a score name is part of the stored uniqueness key.

The evaluator calls a model on every sampled run, so evaluation cost grows with the number of registered evaluators. The model is the judge model, set here or by TraconOnlineEvaluationBuilderExtensions.AddModelRunJudge; registering an evaluator judge without one fails the judge at call time.

Like TraconOnlineEvaluationBuilderExtensions.AddModelRunJudge, this call alone scores nothing: Tracon:OnlineEvaluation:Enabled and its sample rate are still the gate.

builder.AddTracon()
       .AddEvaluatorJudge(
           "relevance",
           new RelevanceEvaluator(),
           options => options.Model = new ModelBinding { Provider = "openai", Model = "gpt-5.4-mini" });

Exceptions

ArgumentNullException

builder or evaluator is null.

ArgumentException

name is not a legal judge name.

AddEvaluatorJudge(ITraconBuilder, string, IEvaluator, Action<ModelRunJudgeOptions>)

Registers a Microsoft.Extensions.AI Evaluation.IEvaluator as a Tracon run judge and configures the shared judge model.

public static ITraconBuilder AddEvaluatorJudge(this ITraconBuilder builder, string name, IEvaluator evaluator, Action<ModelRunJudgeOptions> configure)

Parameters

builder ITraconBuilder

Configuration chain.

name string

The judge name, matching [A-Za-z0-9._-]{1,64}.

evaluator IEvaluator

The evaluator to grade sampled runs with.

configure Action<ModelRunJudgeOptions>

The judge model the evaluator calls.

Returns

ITraconBuilder

The continuation of the chain.

Remarks

The judge model is shared with every other judge: this overload writes the same ModelRunJudgeOptions that TraconOnlineEvaluationBuilderExtensions.AddModelRunJudge writes. Use it to run evaluator judges without also turning on the built-in model judge. Calling it more than once applies each configuration in order, as options configuration always does.

Exceptions

ArgumentNullException

builder, evaluator or configure is null.

ArgumentException

name is not a legal judge name.

AddModelRunJudge(ITraconBuilder, Action<ModelRunJudgeOptions>)

Registers Tracon’s built-in model-based IRunJudge implementation (Tracon.ModelRunJudge).

public static ITraconBuilder AddModelRunJudge(this ITraconBuilder builder, Action<ModelRunJudgeOptions> configure)

Parameters

builder ITraconBuilder

Configuration chain.

configure Action<ModelRunJudgeOptions>

The judge’s model, criteria, and instructions.

Returns

ITraconBuilder

The continuation of the chain.

Remarks

This call ALONE scores nothing. The actual gate for online evaluation is Tracon:OnlineEvaluation:Enabled AND Tracon:OnlineEvaluation:SampleRate (the no-surprises rule: no run is sampled unless both are on). This extension only registers which model is USED as the judge.

Because ModelBinding is a nested type, it is set up in code rather than from a configuration section - the same pattern as other provider bindings (e.g. AgentDefinition.Model).

builder.AddTracon()
       .AddModelRunJudge(options =>
       {
           options.Model = new ModelBinding { Provider = "openai", Model = "gpt-5.4-mini" };
           options.Criteria.Add("Does the response directly answer the question?");
       });

Exceptions

ArgumentNullException

builder or configure is null.