.NET API
EvalStoreContract
Namespace Tracon.Testing.Contracts.Storage · Assembly
Tracon.Testing.Contracts.Xunit.dllBehavior tests for the IEvalStore contract.
public abstract class EvalStoreContract : TenantIsolationContract<IEvalStore>, IAsyncLifetime, IAsyncDisposable- Inheritance
- object ← StoreCancellationContract<IEvalStore> ← TenantIsolationContract<IEvalStore> ← EvalStoreContract
- Implements
- IAsyncLifetime, IAsyncDisposable
- Inherited Members
- TenantIsolationContract<IEvalStore>.TenantA, TenantIsolationContract<IEvalStore>.TenantB, TenantIsolationContract<IEvalStore>.UseAmbientTenant(MutableTenantContext), TenantIsolationContract<IEvalStore>.SeedAsync(string, string, CancellationToken), TenantIsolationContract<IEvalStore>.ExistsAsync(string, object), TenantIsolationContract<IEvalStore>.CountAsync(string, CancellationToken), TenantIsolationContract<IEvalStore>.TryDeleteAsync(string, object), TenantIsolationContract<IEvalStore>.TryOverwriteAsync(string, string), TenantIsolationContract<IEvalStore>.CancellableReadAsync(CancellationToken), TenantIsolationContract<IEvalStore>.CancellableWriteAsync(CancellationToken), TenantIsolationContract<IEvalStore>.WroteAnythingAsync(), TenantIsolationContract<IEvalStore>.Tenant_cannot_read_another_tenants_record(), TenantIsolationContract<IEvalStore>.Tenant_reads_its_own_record(), TenantIsolationContract<IEvalStore>.Tenant_does_not_see_another_tenants_record_in_the_list(), TenantIsolationContract<IEvalStore>.Tenant_cannot_delete_another_tenants_record(), TenantIsolationContract<IEvalStore>.Same_name_lives_independently_across_two_tenants(), TenantIsolationContract<IEvalStore>.AmbientTenant, StoreCancellationContract<IEvalStore>.CreateStoreAsync(), StoreCancellationContract<IEvalStore>.InitializeAsync(), StoreCancellationContract<IEvalStore>.DisposeAsync(), StoreCancellationContract<IEvalStore>.OnDisposeAsync(), StoreCancellationContract<IEvalStore>.CancellableReadAsync(CancellationToken), StoreCancellationContract<IEvalStore>.CancellableWriteAsync(CancellationToken), StoreCancellationContract<IEvalStore>.WroteAnythingAsync(), StoreCancellationContract<IEvalStore>.Canceled_token_throws_on_read(), StoreCancellationContract<IEvalStore>.The_write_this_contract_checks_can_be_read_back(), StoreCancellationContract<IEvalStore>.Canceled_token_throws_on_write_and_leaves_no_trace(), StoreCancellationContract<IEvalStore>.Store, object.GetType(), object.MemberwiseClone(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()
Constructors
Section titled “Constructors”EvalStoreContract()
Section titled “ EvalStoreContract()”protected EvalStoreContract()Properties
Section titled “Properties”QualifiedEvalCasesTable
Section titled “ QualifiedEvalCasesTable”The fully qualified eval_cases table reference this provider’s raw SQL should use.
protected virtual string QualifiedEvalCasesTable { get; }Property Value
Section titled “Property Value”SupportsRawSql
Section titled “ SupportsRawSql”Whether this store’s backing state can be written to directly (SQL providers only).
protected virtual bool SupportsRawSql { get; }Property Value
Section titled “Property Value”Methods
Section titled “Methods”A_row_written_before_expected_tool_calls_existed_reads_back_its_names()
Section titled “ A_row_written_before_expected_tool_calls_existed_reads_back_its_names()”A row written before expected_tool_calls existed holds only the
comma separated names of expected_tools; it reads back as calls
without arguments.
[Fact]public Task A_row_written_before_expected_tool_calls_existed_reads_back_its_names()Returns
Section titled “Returns”The test task.
AddCaseAsync_assigns_seq_atomically()
Section titled “ AddCaseAsync_assigns_seq_atomically()”[Fact]public Task AddCaseAsync_assigns_seq_atomically()Returns
Section titled “Returns”AddCaseAsync_concurrent_promotions_produce_distinct_seq()
Section titled “ AddCaseAsync_concurrent_promotions_produce_distinct_seq()”[Fact]public Task AddCaseAsync_concurrent_promotions_produce_distinct_seq()Returns
Section titled “Returns”AddCaseAsync_does_not_separate_sourceless_cases_from_hand_written_ones()
Section titled “ AddCaseAsync_does_not_separate_sourceless_cases_from_hand_written_ones()”[Fact]public Task AddCaseAsync_does_not_separate_sourceless_cases_from_hand_written_ones()Returns
Section titled “Returns”AddCaseAsync_returns_the_existing_case_the_second_time_for_the_same_source_run_id()
Section titled “ AddCaseAsync_returns_the_existing_case_the_second_time_for_the_same_source_run_id()”[Fact]public Task AddCaseAsync_returns_the_existing_case_the_second_time_for_the_same_source_run_id()Returns
Section titled “Returns”AddCaseAsync_stores_source_fields_and_expected_fields()
Section titled “ AddCaseAsync_stores_source_fields_and_expected_fields()”[Fact]public Task AddCaseAsync_stores_source_fields_and_expected_fields()Returns
Section titled “Returns”AddCaseAsync_stores_the_parameter_values_of_a_promoted_run()
Section titled “ AddCaseAsync_stores_the_parameter_values_of_a_promoted_run()”A promoted case carries the parameter values its source run recorded, and they survive the read back.
[Fact]public Task AddCaseAsync_stores_the_parameter_values_of_a_promoted_run()Returns
Section titled “Returns”The test task.
Remarks
Section titled “Remarks”Without them a promoted case of a parameterized agent fails on a missing value, or runs on a default the production run never used.
Case_result_is_recorded_and_filtered_by_tenant()
Section titled “ Case_result_is_recorded_and_filtered_by_tenant()”[Fact]public Task Case_result_is_recorded_and_filtered_by_tenant()Returns
Section titled “Returns”CountAsync(string, CancellationToken)
Section titled “ CountAsync(string, CancellationToken)”Reports how many records the given tenant sees through the listing endpoint.
protected override ValueTask<int> CountAsync(string tenantId, CancellationToken cancellationToken)Parameters
Section titled “Parameters”tenantId string
The reading tenant.
cancellationToken CancellationToken
The token every store call inside the hook must be handed; see TenantIsolationContract.SeedAsync.
Returns
Section titled “Returns”The number of records seen.
DeleteSuiteAsync_also_deletes_its_cases_and_runs()
Section titled “ DeleteSuiteAsync_also_deletes_its_cases_and_runs()”[Fact]public Task DeleteSuiteAsync_also_deletes_its_cases_and_runs()Returns
Section titled “Returns”DiffRunsAsync_counts_a_case_recorded_twice_once()
Section titled “ DiffRunsAsync_counts_a_case_recorded_twice_once()”[Fact]public Task DiffRunsAsync_counts_a_case_recorded_twice_once()Returns
Section titled “Returns”DiffRunsAsync_refuses_an_unfinished_run()
Section titled “ DiffRunsAsync_refuses_an_unfinished_run()”[Fact]public Task DiffRunsAsync_refuses_an_unfinished_run()Returns
Section titled “Returns”DiffRunsAsync_refuses_two_runs_of_different_suites()
Section titled “ DiffRunsAsync_refuses_two_runs_of_different_suites()”[Fact]public Task DiffRunsAsync_refuses_two_runs_of_different_suites()Returns
Section titled “Returns”DiffRunsAsync_returns_null_for_an_unknown_run()
Section titled “ DiffRunsAsync_returns_null_for_an_unknown_run()”[Fact]public Task DiffRunsAsync_returns_null_for_an_unknown_run()Returns
Section titled “Returns”DiffRunsAsync_returns_null_for_another_tenants_run()
Section titled “ DiffRunsAsync_returns_null_for_another_tenants_run()”[Fact]public Task DiffRunsAsync_returns_null_for_another_tenants_run()Returns
Section titled “Returns”DiffRunsAsync_separates_the_six_buckets()
Section titled “ DiffRunsAsync_separates_the_six_buckets()”[Fact]public Task DiffRunsAsync_separates_the_six_buckets()Returns
Section titled “Returns”DiffRunsAsync_throws_when_retention_removed_only_SOME_of_the_results()
Section titled “ DiffRunsAsync_throws_when_retention_removed_only_SOME_of_the_results()”[Fact]public Task DiffRunsAsync_throws_when_retention_removed_only_SOME_of_the_results()Returns
Section titled “Returns”DiffRunsAsync_throws_when_the_baseline_details_were_removed_by_retention()
Section titled “ DiffRunsAsync_throws_when_the_baseline_details_were_removed_by_retention()”[Fact]public Task DiffRunsAsync_throws_when_the_baseline_details_were_removed_by_retention()Returns
Section titled “Returns”DiffRunsAsync_throws_when_the_candidate_details_were_removed_by_retention()
Section titled “ DiffRunsAsync_throws_when_the_candidate_details_were_removed_by_retention()”[Fact]public Task DiffRunsAsync_throws_when_the_candidate_details_were_removed_by_retention()Returns
Section titled “Returns”ExecuteRawAsync(string)
Section titled “ ExecuteRawAsync(string)”Runs raw SQL against the backing store. Only called when EvalStoreContract.SupportsRawSql is true.
protected virtual ValueTask ExecuteRawAsync(string sql)Parameters
Section titled “Parameters”sql string
The statement.
Returns
Section titled “Returns”A task that completes when the statement has run.
ExistsAsync(string, object)
Section titled “ ExistsAsync(string, object)”Reports whether the record is visible from the given tenant’s viewpoint.
protected override ValueTask<bool> ExistsAsync(string tenantId, object key)Parameters
Section titled “Parameters”tenantId string
The reading tenant.
key object
The key returned by TenantIsolationContract.SeedAsync.
Returns
Section titled “Returns”true if the record is visible.
FormatIdForRawSql(Guid)
Section titled “ FormatIdForRawSql(Guid)”Formats an id for embedding in raw SQL. A provider that stores a Guid as text and compares case-sensitively (SQLite, in uppercase) must override the default (lowercase Guid.ToString).
protected virtual string FormatIdForRawSql(Guid id)Parameters
Section titled “Parameters”id Guid
The id.
Returns
Section titled “Returns”The id as the provider spells it.
GetSuiteAsync_returns_null_for_another_tenant()
Section titled “ GetSuiteAsync_returns_null_for_another_tenant()”[Fact]public Task GetSuiteAsync_returns_null_for_another_tenant()Returns
Section titled “Returns”ListSuitesAsync_returns_only_that_tenant()
Section titled “ ListSuitesAsync_returns_only_that_tenant()”[Fact]public Task ListSuitesAsync_returns_only_that_tenant()Returns
Section titled “Returns”QueryRunsAsync_filters_by_suite_and_puts_the_newest_first()
Section titled “ QueryRunsAsync_filters_by_suite_and_puts_the_newest_first()”[Fact]public Task QueryRunsAsync_filters_by_suite_and_puts_the_newest_first()Returns
Section titled “Returns”ReplaceCasesAsync_assigns_sequence_numbers_and_replaces_previous_cases()
Section titled “ ReplaceCasesAsync_assigns_sequence_numbers_and_replaces_previous_cases()”[Fact]public Task ReplaceCasesAsync_assigns_sequence_numbers_and_replaces_previous_cases()Returns
Section titled “Returns”ReplaceCasesAsync_keeps_an_identifier_it_was_given()
Section titled “ ReplaceCasesAsync_keeps_an_identifier_it_was_given()”[Fact]public Task ReplaceCasesAsync_keeps_an_identifier_it_was_given()Returns
Section titled “Returns”ReplaceCasesAsync_keeps_the_promotion_fields_it_was_given()
Section titled “ ReplaceCasesAsync_keeps_the_promotion_fields_it_was_given()”[Fact]public Task ReplaceCasesAsync_keeps_the_promotion_fields_it_was_given()Returns
Section titled “Returns”ReplaceCasesAsync_leaves_parameters_null_for_a_case_that_declares_none()
Section titled “ ReplaceCasesAsync_leaves_parameters_null_for_a_case_that_declares_none()”[Fact]public Task ReplaceCasesAsync_leaves_parameters_null_for_a_case_that_declares_none()Returns
Section titled “Returns”ReplaceCasesAsync_reads_back_an_empty_list_for_a_case_that_expects_no_tool_call()
Section titled “ ReplaceCasesAsync_reads_back_an_empty_list_for_a_case_that_expects_no_tool_call()”A case that expects no tool call reads back an empty list, never null.
[Fact]public Task ReplaceCasesAsync_reads_back_an_empty_list_for_a_case_that_expects_no_tool_call()Returns
Section titled “Returns”The test task.
ReplaceCasesAsync_round_trips_an_expected_argument_of_every_json_kind()
Section titled “ ReplaceCasesAsync_round_trips_an_expected_argument_of_every_json_kind()”An argument of every JSON kind survives the round trip, and a call without arguments stays distinct from one with an empty set.
[Fact]public Task ReplaceCasesAsync_round_trips_an_expected_argument_of_every_json_kind()Returns
Section titled “Returns”The test task.
Remarks
Section titled “Remarks”The column is jsonb in PostgreSQL and text elsewhere. jsonb
normalizes number spelling and object member order, so the comparison is
by JSON meaning; an argument that comes back as a different KIND (a
number as a string, a nested object as text) is the failure this guards.
ReplaceCasesAsync_stores_expected_tool_calls_and_context()
Section titled “ ReplaceCasesAsync_stores_expected_tool_calls_and_context()”[Fact]public Task ReplaceCasesAsync_stores_expected_tool_calls_and_context()Returns
Section titled “Returns”ReplaceCasesAsync_stores_parameter_values_for_a_parameterized_agent()
Section titled “ ReplaceCasesAsync_stores_parameter_values_for_a_parameterized_agent()”[Fact]public Task ReplaceCasesAsync_stores_parameter_values_for_a_parameterized_agent()Returns
Section titled “Returns”Run_lifecycle_transitions_from_running_to_completed()
Section titled “ Run_lifecycle_transitions_from_running_to_completed()”[Fact]public Task Run_lifecycle_transitions_from_running_to_completed()Returns
Section titled “Returns”Run_reads_do_not_leak_across_tenants()
Section titled “ Run_reads_do_not_leak_across_tenants()”[Fact]public Task Run_reads_do_not_leak_across_tenants()Returns
Section titled “Returns”SaveSuiteAsync_assigns_an_id_to_a_new_suite()
Section titled “ SaveSuiteAsync_assigns_an_id_to_a_new_suite()”[Fact]public Task SaveSuiteAsync_assigns_an_id_to_a_new_suite()Returns
Section titled “Returns”SaveSuiteAsync_keeps_the_id_for_the_same_name_and_updates()
Section titled “ SaveSuiteAsync_keeps_the_id_for_the_same_name_and_updates()”[Fact]public Task SaveSuiteAsync_keeps_the_id_for_the_same_name_and_updates()Returns
Section titled “Returns”SaveSuiteAsync_round_trips_a_suite_that_carries_no_checks()
Section titled “ SaveSuiteAsync_round_trips_a_suite_that_carries_no_checks()”A suite saved with no checks round-trips through every provider.
[Fact]public Task SaveSuiteAsync_round_trips_a_suite_that_carries_no_checks()Returns
Section titled “Returns”Remarks
Section titled “Remarks”The eval side of the same class as the schedule payload: the unset
value used to reach the column as the literal null, which SQL
Server’s CHECK (ISJSON(checks) = 1) rejects.
SeedAsync(string, string, CancellationToken)
Section titled “ SeedAsync(string, string, CancellationToken)”Writes a sample record for the given tenant.
protected override ValueTask<object> SeedAsync(string tenantId, string name, CancellationToken cancellationToken)Parameters
Section titled “Parameters”tenantId string
The tenant that owns the record.
name string
The name distinguishing the record. The same name may be used by two tenants.
cancellationToken CancellationToken
The token every store call inside the hook must be handed. The cancellation contract calls this hook with a token that is already cancelled, so a hook that drops the token reports a store as sound when it is not.
Returns
Section titled “Returns”The key to use for reading the record back.
TryDeleteAsync(string, object)
Section titled “ TryDeleteAsync(string, object)”Attempts to delete the record on behalf of the given tenant.
protected override ValueTask<bool?> TryDeleteAsync(string tenantId, object key)Parameters
Section titled “Parameters”tenantId string
The tenant attempting the delete.
key object
The key returned by TenantIsolationContract.SeedAsync.
Returns
Section titled “Returns”true if the delete succeeded, false if the record was not found; null if the store does not offer deletion.