EvalRunDiffBuilder

Namespace Tracon · Assembly Tracon.Abstractions.dll

Aligns two eval runs’ per-case results into an EvalRunDiff.

public static class EvalRunDiffBuilder

Inheritance

object ← EvalRunDiffBuilder

Inherited Members

object.GetType(), object.MemberwiseClone(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()

Remarks

This is the whole comparison policy — the bucket rules, the repetition rule, the ordering and the three refusals — in one place, so that every IEvalStore implementation shares it instead of rewriting it. A store’s IEvalStore.DiffRunsAsync reads the two runs and their case results and hands them here.

Methods

Build(EvalRun, EvalRun, IReadOnlyList<EvalCaseResult>, IReadOnlyList<EvalCaseResult>, int, int)

Aligns two completed runs of the same suite, case by case.

public static EvalRunDiff Build(EvalRun baseline, EvalRun candidate, IReadOnlyList<EvalCaseResult> baselineResults, IReadOnlyList<EvalCaseResult> candidateResults, int skip = 0, int take = 2147483647)

Parameters

baseline EvalRun

The run being compared against.

candidate EvalRun

The run being judged.

baselineResults IReadOnlyList<EvalCaseResult>

The baseline run’s per-case results.

candidateResults IReadOnlyList<EvalCaseResult>

The candidate run’s per-case results.

skip int

The number of aligned cases to skip.

take int

The maximum number of aligned cases to return.

Returns

EvalRunDiff

The difference between the two runs.

Remarks

A case that carries several result rows within one run — the suite ran with repetitions, or the job was redelivered and recorded its results a second time (IJobHandler is at-least-once) — counts as passed only when every one of its rows passed. That is the same rule EvalRun.Passed itself uses for repetitions, and it fails safe for the redelivery case: a stale failing row can show a case as broken when it is not, but it can never hide a break.

Exceptions

EvalRunDiffUnavailableException

The two runs measure different suites, one of them has not completed, or one of them holds results for fewer cases than its summary counts - the rest were removed by retention, which deletes in batches and can leave a run partly trimmed. Never answered with a shortened diff: reporting only the surviving cases reads as “nothing changed” for the rest.