IRunJudge

Namespace Tracon · Assembly Tracon.Abstractions.dll

The extension point that scores a completed production run.

public interface IRunJudge

Remarks

Implementations are singleton services. The same instance can receive concurrent calls, so it must be thread-safe and must not keep per-run state in instance fields.

A call can still be repeated for the same run: once when another judge causes the whole job to retry (a judge that already wrote a score for this run is not called again on that retry — only a judge that has not yet scored the run runs), and always when a caller re-triggers manual scoring via POST /api/runs/{id}/judge. Implementations must make side effects idempotent. IRunJudge.Name is a stable low-cardinality identifier. It must match [A-Za-z0-9._-]{1,64} and is used in metric tags and score fields.

The supplied cancellation token is the call budget. Do not throw an OperationCanceledException for an implementation-owned timeout. Tracon applies its own timeout to each call.

RunJudgeContext.TenantId is authoritative. A judge that starts a Tracon run must set its run kind to RunKind.Eval. A model-backed judge uses IModelProviderRegistry.CreateSetupChatClientAsync.

Properties

Name

The stable judge name, written as judge:{Name} in score fields.

string Name { get; }

Property Value

string

Methods

JudgeAsync(RunJudgeContext, CancellationToken)

Scores the run.

ValueTask<RunJudgment> JudgeAsync(RunJudgeContext context, CancellationToken cancellationToken = default)

Parameters

context RunJudgeContext

The context the judge sees.

cancellationToken CancellationToken

The cancellation token.

Returns

ValueTask<RunJudgment>

The judge’s verdict.