.NET API
OnlineEvalSummaryService
Tracon.Core.dllMaintains a sliding-window summary of online evaluation scores and publishes WebhookEvents.RunScoreLow when its threshold is crossed.
public sealed class OnlineEvalSummaryServiceInheritance
Section titled “Inheritance”object ← OnlineEvalSummaryService
Inherited Members
Section titled “Inherited Members”object.GetType(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()
Remarks
Section titled “Remarks”The score window is in memory, with one queue per tenant,
and resets when the process restarts. This service is only an inexpensive
live indicator and alarm; for an authoritative result that survives a
restart, query IRunScoreStore.SummarizeAsync instead
(GET /api/evaluation/scores/summary). The same live-indicator design
is used for RunSampler’s hourly budget (the no-surprises rule: no durable counter store).
One low score does not produce an alarm. Threshold evaluation does not run before OnlineEvaluationOptions.MinSampleSize is met. Models are noisy, and otherwise notifications quickly become ignored.
Constructors
Section titled “Constructors”OnlineEvalSummaryService(IRunStore, IOptionsMonitor<OnlineEvaluationOptions>, IWebhookPublisher?, TimeProvider?)
Section titled “ OnlineEvalSummaryService(IRunStore, IOptionsMonitor<OnlineEvaluationOptions>, IWebhookPublisher?, TimeProvider?)”Maintains a sliding-window summary of online evaluation scores and publishes WebhookEvents.RunScoreLow when its threshold is crossed.
public OnlineEvalSummaryService(IRunStore runStore, IOptionsMonitor<OnlineEvaluationOptions> optionsMonitor, IWebhookPublisher? webhookPublisher = null, TimeProvider? timeProvider = null)Parameters
Section titled “Parameters”runStore IRunStore
optionsMonitor IOptionsMonitor<OnlineEvaluationOptions>
webhookPublisher IWebhookPublisher?
timeProvider TimeProvider?
Remarks
Section titled “Remarks”The score window is in memory, with one queue per tenant,
and resets when the process restarts. This service is only an inexpensive
live indicator and alarm; for an authoritative result that survives a
restart, query IRunScoreStore.SummarizeAsync instead
(GET /api/evaluation/scores/summary). The same live-indicator design
is used for RunSampler’s hourly budget (the no-surprises rule: no durable counter store).
One low score does not produce an alarm. Threshold evaluation does not run before OnlineEvaluationOptions.MinSampleSize is met. Models are noisy, and otherwise notifications quickly become ignored.
Methods
Section titled “Methods”GetSummaryAsync(string, CancellationToken)
Section titled “ GetSummaryAsync(string, CancellationToken)”Returns a tenant’s current window summary and judge cost.
public ValueTask<OnlineEvaluationSummary> GetSummaryAsync(string tenantId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<OnlineEvaluationSummary>
RecordScoreAsync(string, double, CancellationToken)
Section titled “ RecordScoreAsync(string, double, CancellationToken)”Adds a newly written judge score to the window and publishes WebhookEvents.RunScoreLow when needed.
public ValueTask RecordScoreAsync(string tenantId, double score, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
score double
The written score, from 0 to 100.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”Remarks
Section titled “Remarks”Only a judge’s headline score reaches this window: the one whose name equals the judge’s own. The window and its alarm are defined on the 0-100 scale, so a bridged metric on another scale is stored but never averaged in. See JudgeScore.