# Voice and live conversation > Tracon documentation. Tracon is a .NET package family that adds a control > plane on Microsoft Agent Framework, published as NuGet packages that run > inside your own ASP.NET Core application. > > Page: https://tracon.dev/guides/voice/ > Summary: Add speech tools and an opt-in realtime WebSocket conversation with explicit formats, limits, cost, privacy, and deployment rules. :::note[Preview packages] Tracon is published as `1.0.0-preview.6`. Use `--prerelease` for discovery or pin the exact version for reproducible builds. ::: Tracon has three voice layers: | Layer | Use it for | Registration | |---|---|---| | Voice tools and REST | Generate speech, transcribe an attachment, list voices, and voice a multi-speaker dialogue from a workflow node | `UseVoice()` from `Tracon.Voice` | | Live conversation | Keep a bidirectional audio/text session over WebSocket, with Tracon transcribing and synthesizing | `UseVoiceConversation()` from Core | | Provider-hosted live voice | Let a realtime model run the conversation and hand the heavy work back to your agents | `UseLiveVoice()` plus a provider such as `UseOpenAILive()` | The second and third solve the same problem in opposite directions, and the choice matters. Under `UseVoiceConversation()` the audio reaches Tracon, so every turn is a run, the content guard sees what was said, and any speech provider works. Under `UseLiveVoice()` the provider owns the conversation and the audio never touches your server — you get true full duplex and provider-side turn detection, and you give up seeing most of what was said. [What you give up](/guides/live-voice/#what-the-provider-hosted-path-gives-up) lists that trade in full. ```mermaid flowchart LR accTitle: The two voice layers and their hosting cost accDescr: UseVoice adds request and response speech tools over the ordinary HTTP surface. UseVoiceConversation additionally maps a WebSocket route and binds a socket to one instance for the length of the conversation, which is why it is opt-in. subgraph Tools["UseVoice — request/response"] SYN["Synthesize speech"] TRN["Transcribe an attachment"] LST["List voices"] end subgraph Live["UseVoiceConversation — opt-in"] WS["WebSocket /api/voice/sessions/id/stream
subprotocol tracon.voice.v1"] BOUND["Socket bound to one instance
for minutes"] end Tools --> HTTP["Ordinary HTTP surface"] WS --> BOUND BOUND --> LIMITS["Per-tenant connection, duration,
idle, and utterance limits"] ``` The tool layer is request/response. The conversation layer changes the hosting model: a socket stays bound to one application instance for minutes. Enable only the layer you need. ## Add speech tools Install `Tracon.Voice`, keep the provider key out of files, and register it: ```bash dotnet add package Tracon.Voice --prerelease dotnet user-secrets set "Tracon:Voice:ApiKey" "..." ``` ```csharp var tracon = builder.AddTracon() .UseVoice(builder.Configuration.GetSection(VoiceOptions.SectionName)); tracon.AddAgent(new AgentDefinition { Name = "voice-assistant", Instructions = "Answer clearly. Speak the answer when the caller asks.", Model = new ModelBinding { Provider = OpenAIProviderNames.ChatCompletions, Model = "your-model", }, ToolNames = ["speak", "transcribe", "list_voices"], }); ``` | Tool | Input and result | |---|---| | `speak` | Text in; generated audio attachment ID out | | `transcribe` | Audio attachment ID in; resolved text out | | `list_voices` | Provider voice names, identifiers, and safe scalar attributes out | Generated audio is stored through the normal attachment contract. The model receives an attachment ID, not a large byte array. The file name takes its extension from the type the attachment guard detects — `.mp3`, `.ogg` or `.wav` — not from the requested format. `UseVoice()` validates `Tracon:Voice` when the application starts. A missing key, an unknown `Provider` or a headerless `OutputFormat` stops the host before the first request, not at the first tool call. ### Voice attributes `VoiceDescriptor.Attributes` carries a provider's own bounded, safe scalar metadata for a voice — for example `gender`, `language`, `accent`, `age`, or `use-case`. `VoiceAttributeNames` names the well-known keys, but the set is not closed: any other safe scalar label a provider reports survives under its own key. The collection is empty, never `null`, when the provider reports nothing. ```csharp var voices = await synthesizer.ListVoicesAsync(); var femaleEnglish = voices.Where(v => v.Attributes.GetValueOrDefault(VoiceAttributeNames.Gender) == "female" && (v.Attributes.GetValueOrDefault(VoiceAttributeNames.Language) ?? "").Contains("en")); ``` Bounds keep a misbehaving provider response from becoming a payload: at most 32 attributes per voice, a 64-character key, a 256-character value. A duplicate key that differs only by case or by `_`/`-` collapses to one canonical lowercase, hyphenated entry (`use_case` and `use-case` both become `use-case`). `preview_url` is never carried — playing it would connect the browser directly to the provider's address — and neither is any other provider-internal field; only the declared safe scalar labels reach the model. Whether `language` is populated depends entirely on what the underlying provider reports: the built-in ElevenLabs implementation reads it from that provider's per-model verified-languages list, joining multiple verified languages into one comma-separated value (for example `"en,fr"`). A provider that reports nothing comparable simply omits the key — it is never guessed. ### Use a storable output format The attachment guard trusts magic bytes, not a claimed media type. Raw `pcm_*` and `ulaw_*` output has no file header and cannot be stored as an attachment. The default `mp3_44100_128` is valid; choosing a non-storable provider format fails option validation at startup. ### Voice cost has a different unit Synthesis cost uses characters and transcription cost uses duration. Tracon writes that measurement on the tool invocation, with the provider and model that did the work, and never adds it to the token cost: a character, a second and a token are different units, so a quantity is never added across them. The *amount* is a different matter. It joins the run's non-token total (`nonTokenCost` on the run), which the call-tree budget, the tenant's cost quota and the `run.completed` webhook count beside the token cost. See [Report what a tool spent beyond tokens](/concepts/tools/#report-what-a-tool-spent-beyond-tokens) for the rule, and [Report what a node spent beyond tokens](/concepts/workflow-function-nodes/#report-what-a-node-spent-beyond-tokens) for a workflow function node that voices text itself. If pricing is not configured, cost stays `null`: the measurement is listed as unpriced and left out of the total, because zero would claim the call was free. If the provider omits billed characters, the text length is used and marked as an estimate; an estimated amount still adds, and the total says it rests on an estimate. Prices live under `Tracon:Pricing:Voice:{provider}:{model}`. `{provider}` is the `ProviderName` of the implementation that did the work — `elevenlabs` for the built-in client — not the `Tracon:Voice:Provider` setting. `{model}` is the model the implementation reports on its result (`SpeechAudio.ModelId`, `SpeechTranscript.ModelId`), then the one the request named. Only when neither is known does the default provider fall back to `SynthesisModelId` or `TranscriptionModelId`, and only the built-in client to its own default model. The built-in client always reports the model it used. ### Character-level timing `POST /api/voice/speak` accepts `includeTimestamps` (default `false`). When set, the response's `alignment` field carries one entry per character — the character itself and its start/end offset from the beginning of the audio. Alignment is not a separate billing unit: `characters` and `cost` are computed exactly as without it. Timestamps are not available on the streaming synthesis path (`ISpeechSynthesizer.SynthesizeStreamingAsync`): that method returns raw audio chunks only and has no channel to carry per-character timing back to the caller. Requesting both together throws rather than silently dropping the alignment. Grouping characters into words, or producing a subtitle format such as SRT or VTT, is left to the consumer — the raw per-character form is what the provider gives, and turning it into something else is application-specific. ### Use your own speech provider The built-in implementation uses ElevenLabs through raw `HttpClient` and source-generated JSON. The package adds no NuGet dependency and remains AOT compatible. To replace it, register your implementation **before** `UseVoice()`; `TryAdd` preserves it: ```csharp builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.AddTracon().UseVoice(options => { options.DefaultVoiceId = "customer-care"; options.SynthesisModelId = "contoso-tts-1"; }); ``` With both contracts registered first, no API key is needed. `Provider`, `ApiKey` and `Endpoint` configure only the built-in client, so they are neither required nor validated. The other settings still apply and are still validated at startup: the `speak` tool passes `OutputFormat` and `SynthesisModelId` to your synthesizer, and `transcribe` passes `TranscriptionModelId` to your transcriber. Replace only one contract and the built-in client serves the other — the key is then still required. `UseVoice()` reads the service collection when you call it. A registration added after the call wins resolution but not validation: the host stops at startup with a message that says to register first. Three behaviors follow the bound implementation, not the settings: | Behavior | Read from | |---|---| | Character limit of `speak` and `POST /api/voice/speak` | `ISpeechSynthesizer.MaxCharactersPerRequest` | | Price lookup | `Tracon:Pricing:Voice:{ProviderName}:{model}` of the synthesizer or transcriber that did the work | | `GET /api/voice/health` | Your synthesizer's own `IVoiceHealthCheck`, when it implements one; otherwise a voice-list call that reports your `ProviderName` | With no model identifier set, your provider's cost stays `null`: Tracon does not borrow the built-in client's default model or its price. A provider that already ships a `Microsoft.Extensions.AI` speech client needs no hand-written adapter. See [Use any Microsoft.Extensions.AI speech client](#use-any-microsoftextensionsai-speech-client). ### Several speech providers in one host A host that picks the text-to-speech engine per pipeline or per customer — a hosted service for one, a model on its own GPU for another — registers each engine under a name, next to the default one: ```csharp builder.AddTracon() .UseVoice(builder.Configuration.GetSection(VoiceOptions.SectionName)) .AddSpeechSynthesizer("studio", provider => new StudioSpeechSynthesizer( provider.GetRequiredService()["Studio:ApiKey"])) .AddSpeechSynthesizer("gpu", _ => new LocalGpuSynthesizer(new Uri("http://gpu-01:9000/"))) .AddSpeechTranscriber("gpu", _ => new LocalGpuTranscriber(new Uri("http://gpu-01:9000/"))); ``` A name is 1 to 64 characters of `a-z`, `0-9`, `.`, `_` and `-`, compared in lower case; synthesizer and transcriber names are separate. Registering the same name twice throws at startup instead of letting the later call win. The call can come before or after `UseVoice()`, and it never changes which provider `UseVoice()` binds as the default. A factory that returns one instance for two names shares that instance — and its concurrency limit — between them. A caller selects an engine by name: | Caller | How it names the engine | |---|---| | `POST /api/voice/speak` | `"provider": "studio"` in the body | | `GET /api/voice/voices` | `?provider=studio` | | A workflow function node | `ISpeechProviderResolver.GetSynthesizerAsync(name)` with a name from the node's settings or a run parameter | | The `speak`, `transcribe` and `list_voices` tools, the conversation socket | Always the default provider | `GET /api/voice/providers` lists every registration — its name (`null` for the default), its kind, the provider name it reports, whether it accepts a tenant's own key, and `dialogue`: the limits of one dialogue request, or `null` when the provider cannot voice a dialogue. An unknown name answers `400` and names the registered ones; a request without a name when no default is registered answers `501`. A provider failure the speech adapter reports as a `TraconException` (the built-in adapters do) answers `502`, while it speaks or lists its voices; any other exception is a server error. A function node takes the resolver when it is built, because the service provider a node sees at call time is not the host's: ```csharp builder.AddTracon() .UseWorkflows() .AddWorkflowFunction, List, NarrationSettings>( "narrate", provider => { var speech = provider.GetRequiredService(); var pricing = provider.GetRequiredService(); return async (messages, settings, context, cancellationToken) => { var synthesizer = await speech.GetSynthesizerAsync(settings.Provider, cancellationToken); var audio = await synthesizer.SynthesizeAsync( new SpeechRequest { Text = messages[^1].Text }, cancellationToken); // The node called the engine itself, so Tracon did not see the spend. var characters = audio.CharactersBilled ?? messages[^1].Text.Length; await context.ReportUsageAsync(new ToolCallUsage { Unit = ToolUsageUnits.Characters, Quantity = characters, Cost = pricing.ForSynthesis(synthesizer.ProviderName, audio.ModelId, characters), Currency = pricing.Currency, IsEstimated = audio.UsageSource != SpeechUsageSource.Provider, Provider = synthesizer.ProviderName, Model = audio.ModelId, }); return [new ChatMessage(ChatRole.Assistant, $"{audio.Data.Length} bytes")]; }; }); ``` A named engine never receives the default's `SynthesisModelId` or `OutputFormat`: a model identifier of one engine sent to another is a wrong request. Set them on the request when the engine needs them. `IVoicePricingReader.ForSynthesis` returns `null` when no price is configured; the item is then listed as unpriced instead of counting as zero. ### Use any Microsoft.Extensions.AI speech client A provider that already ships an `ITextToSpeechClient` or `ISpeechToTextClient` needs no hand-written adapter. Register the client in the container with the extension of its own package, then bind it to a name. Tracon serves it through its own speech contracts, so it passes the same tenant checks as every other engine: ```csharp #pragma warning disable MEAI001 // Microsoft.Extensions.AI marks its speech clients experimental builder.Services.AddKeyedTextToSpeechClient("studio", provider => new StudioTextToSpeechClient(provider.GetRequiredService()["Studio:ApiKey"])); builder.Services.AddKeyedSpeechToTextClient("studio", provider => new StudioSpeechToTextClient(provider.GetRequiredService()["Studio:ApiKey"])); builder.AddTracon() .AddSpeechSynthesizer("studio", new TextToSpeechClientBinding { ServiceKey = "studio", ProviderName = "studio", MaxCharactersPerRequest = 4000, Voices = [new VoiceDescriptor { VoiceId = "river", Name = "River" }], }) .AddSpeechTranscriber("studio", new SpeechToTextClientBinding { ServiceKey = "studio", ProviderName = "studio", }); ``` The binding says what the client cannot say about itself: | Field | Meaning | |---|---| | `ServiceKey` | The key the client is registered under. Leave it out for the unkeyed registration. | | `ProviderName` | The name a tenant's egress policy and provider bindings use. Leave it out to read the name from the client's metadata. | | `MaxCharactersPerRequest` | Required, greater than zero. `speak` and `POST /api/voice/speak` refuse longer text before the client is called. | | `Voices` | The list `GET /api/voice/voices` returns. The client has no way to list its voices, so the list is a declaration. Empty by default. | What the bridge does and does not do: - **The host stops at startup** when the client is not registered, or when neither the binding nor the client names a provider. A provider with no name would pass every egress policy. - **A tenant with its own key for the provider is refused** with `403`. The client was built with your key and cannot take another. - **Billing is an estimate.** The client reports no billed characters, so Tracon records the length of the text and marks it as estimated. A transcription has a duration only when the client reports a start and an end time; without one it has no cost. - **Character timing and dialogue are refused.** `IncludeTimestamps` fails, and `GetDialogueSynthesizerAsync` reports that the engine cannot voice a dialogue. - **The media type is not passed on.** The client takes a stream and no media type. - **A failure is a `502`** that carries the exception type and, for HTTP, the status code. The client's own message can echo the request, so it stays in the log. - **Request settings reach the client unchanged,** as its `AdditionalProperties`. Tracon cannot tell which keys the client reads, so it cannot refuse an unknown one. Put no secret there: `LoggingTextToSpeechClient` writes its options to the log at Trace level. No Tracon public member names an experimental type: the bindings hold a key, a name, a limit, and a list. ### OpenAI speech `UseOpenAISpeech` registers OpenAI text-to-speech and speech-to-text as named providers. It shares the client of `UseOpenAI`, so call that first: ```csharp builder.AddTracon() .UseOpenAI(builder.Configuration["OpenAI:ApiKey"]!) .UseOpenAISpeech(options => { options.SynthesisModel = "gpt-4o-mini-tts"; options.TranscriptionModel = "gpt-4o-mini-transcribe"; }); ``` ```http POST /tracon/api/voice/speak { "text": "Your order is on its way.", "provider": "openai" } ``` Set at least one model on `OpenAISpeechOptions`. The provider name is `openai`, the name the chat provider uses, so one egress rule covers chat, images, and speech. | Option | Default | Meaning | |---|---|---| | `SynthesisModel` | none | The text-to-speech model. A synthesizer is registered when it is set. | | `TranscriptionModel` | none | The speech-to-text model. A transcriber is registered when it is set. | | `SynthesizerName`, `TranscriberName` | `openai` | The registration names. Register a second model under another name. | | `MaxCharactersPerRequest` | `4096` | The limit `speak` enforces. | | `Voices` | empty | The voices `GET /api/voice/voices` lists. A call that names none is spoken in `alloy`. | | `UseAsDefault` | `false` | Also register the synthesizer and the transcriber as the unnamed defaults, with `TryAdd`. Call `UseOpenAISpeech` before `UseVoice` for it to take effect. | Rules that follow from the OpenAI service: - **One model for each name.** A request that names another model is refused instead of being answered by one model and priced as another. Leave the model out of the request. - **Format names are read as containers.** `speak` sends the configured `OutputFormat`, an ElevenLabs name such as `mp3_44100_128`. OpenAI names the container only, so the part before the first underscore is the format: `mp3`, `opus`, `aac`, `flac`, `wav`, or `pcm`. - **Streaming passes the audio on as the service sends it,** chunked and not as server-sent events, so playback can start early and every speech model serves it. - **Provider settings are refused.** OpenAI speech reads none. - **A transcription has no cost.** The answer carries no duration. The file name sent to the service follows the audio bytes, so a WAV stream is sent as a WAV file. - **The project needs the model.** Without access the call fails with `OpenAI speech request failed with HTTP 403 (model_not_found)`: status and error code, never the service's text. - **Native AOT.** The package publishes with Native AOT, including this path. Gemini speech has [its own page](/guides/gemini-speech/). ### Voice a dialogue with several speakers A two-host podcast voiced one line at a time and joined together sounds read aloud. A provider that voices the whole exchange in one request keeps the turn-taking natural. `ISpeechDialogueSynthesizer` is that capability: an ordered list of turns, each with its own voice, in one request and one audio result. The built-in ElevenLabs client implements it; a synthesizer of your own implements it next to `ISpeechSynthesizer`. Code takes the capability from the resolver, by the same name a synthesizer is registered under. Nothing registers it in the container: a direct registration would reach the provider without the tenant's checks. ```csharp builder.AddTracon() .UseVoice(builder.Configuration.GetSection(VoiceOptions.SectionName)) .UseWorkflows() .AddWorkflowFunction, List, NarrationSettings>( "two-voices", provider => { var speech = provider.GetRequiredService(); return async (messages, settings, context, cancellationToken) => { var dialogue = await speech.GetDialogueSynthesizerAsync(settings.Provider, cancellationToken); var result = await dialogue.SynthesizeDialogueAsync( new SpeechDialogueRequest { Turns = [ new SpeechDialogueTurn { Text = "Welcome back.", VoiceId = "host-voice" }, new SpeechDialogueTurn { Text = "Glad to be here.", VoiceId = "guest-voice" }, ], OutputFormat = "pcm_24000", IncludeTimestamps = true, }, cancellationToken); return [new ChatMessage(ChatRole.Assistant, $"{result.Segments?.Count ?? 0} segments")]; }; }); ``` | Rule | What happens | |---|---| | Limits | `DialogueCapabilities` states the most distinct voices (`MaxVoices`) and characters (`MaxCharactersPerRequest`) one request takes. The built-in client takes 10 voices and `Tracon:Voice:MaxDialogueCharactersPerRequest` characters (default 2,000, the provider's guidance). A request over a limit, an empty turn or a turn without a voice is refused before the call — Tracon does not split or cut a dialogue | | Splitting and joining | Yours. Split a long script where its content allows, voice each part, and join the audio. Raw `pcm_*` output has no file header: join the samples, then add the header | | Model | The request's `ModelId`, then `Tracon:Voice:DialogueModelId`, then the provider's default. The single-voice `SynthesisModelId` is never used. With the provider's default, the result names no model, so a price keyed by model is not found | | Timing | `IncludeTimestamps` returns `Segments` — the turn index, voice and start and end of each part — and character `Alignment` over the turns' texts joined in order. A report that names a turn or voice the request did not have, leaves a turn without a segment, or is out of order gives `Segments = null` and keeps the audio | | Usage | `Audio.CharactersBilled` comes from the provider when it reports a count (`UsageSource.Provider`), otherwise from the turns' text (`Estimated`). `Audio.ProviderRequestId` is the provider's id for the request | | Tenant rules | The same as every other speech call (next section), applied to the dialogue capability's `ProviderName`. A synthesizer without the capability fails the call with a message that names it; the built-in client is not used in its place | There is no dialogue tool and no dialogue HTTP endpoint: a dialogue is a workflow step, written in code. ### Provider request settings `SpeechRequest.ProviderSettings` and `SpeechDialogueRequest.ProviderSettings` carry a provider's own tuning, keyed under its prefix, in the same form a model binding's provider settings take. The built-in client reads these keys (`ElevenLabsProviderNames` holds them as constants): | Key | Value | Single voice | Dialogue | |---|---|---|---| | `elevenlabs.stability` | number from 0 to 1 | `voice_settings.stability` | `settings.stability` | | `elevenlabs.similarityBoost` | number from 0 to 1 | `voice_settings.similarity_boost` | `settings.similarity` | | `elevenlabs.style` | number | `voice_settings.style` | refused | | `elevenlabs.useSpeakerBoost` | boolean | `voice_settings.use_speaker_boost` | refused | | `elevenlabs.speed` | number, 1 is normal | `voice_settings.speed` | refused | | `elevenlabs.seed` | integer from 0 to 4294967295 | `seed` | `seed` | | `elevenlabs.previousText` | text | `previous_text` | `previous_text`, at most 100 characters | | `elevenlabs.nextText` | text | `next_text` | `future_text`, at most 100 characters | | `elevenlabs.previousRequestIds` | list of at most 3 texts | `previous_request_ids` | `previous_request_ids` | An unknown key, another provider's key, a value of the wrong type or outside its range, and on a dialogue a key the dialogue service does not take, are refused before the call. The message names the key and the accepted values, never the value itself: a value can be your user's text. The provider's dialogue service accepts an unknown field without an error, so a silently dropped setting would never show — that is why the client refuses instead. The seed is a best effort: two requests with the same seed can return different audio. To keep the speech continuous across parts of a long script, pass each earlier result's `Audio.ProviderRequestId` in `elevenlabs.previousRequestIds` (the closest one last). Add an id only when it is not `null`; a list with a missing id is refused. The `speak` tool and `POST /api/voice/speak` send no provider settings. A synthesizer of your own checks its keys with the same rule: ```csharp ModelProviderSettings.Validate(request.ProviderSettings, "contoso", ["contoso.pitch"]); ``` ### Tenant policy and credentials for speech Every speech call obeys the calling tenant's provider rules, the same rules a model call follows ([egress and per-tenant credentials](/concepts/governance/)). The checks run before any text or audio leaves the process: - The tenant's egress policy must allow the provider name the engine **reports** (`ProviderName`), not the name it was registered under — two names for one service are one provider to the policy. A local engine is not exempt: what it does with the text is outside Tracon's view. Otherwise the call is refused: `403` on the HTTP endpoints and on the conversation handshake, an error result for a tool, a failed node for a workflow — a refusal is not retried. - A tenant with its own binding for that provider name is billed on its own key. The key reaches only an implementation that sets `SupportsTenantCredentials` to `true`; it arrives in `SpeechRequest.Credential`, `SpeechTranscriptionOptions.Credential`, `SpeechDialogueRequest.Credential` and `ListVoicesWithCredentialAsync`. Any other implementation is refused for that tenant rather than called on its own configured key — the wrong account. The built-in ElevenLabs client does not take a tenant key, and a request that carries one directly fails with `NotSupportedException` instead of using the configured key. - A binding whose configuration key has no value, or that sets an endpoint override, is refused; the call never falls back to the shared key. The tenant is the current run's tenant, or the request's tenant outside a run, so a queued workflow is judged by the tenant that queued it. The policy and the binding are read each time a provider is resolved. A tool call, an HTTP request and a function node resolve on every call, so a policy saved now stops their next call. The conversation socket resolves its transcriber and synthesizer **once, when the connection opens**: an open connection keeps them until it closes, and a policy saved meanwhile applies from the next connection — close the connection to apply it at once. Reading the provider list and `GET /api/voice/health` apply no tenant check; the health check calls the default synthesizer directly as a platform check, with the configured key and no tenant data. :::caution[Behavior change] A tenant with a saved egress policy that does not name its speech provider can no longer synthesize, transcribe or list voices. Add the provider name (`elevenlabs` for the built-in client) to the policy — the tenant panel in the console lists speech providers next to the model providers. ::: ## Enable live conversation Live conversation needs both an `ISpeechTranscriber` and `ISpeechSynthesizer`. Add the opt-in before the application is built: ```csharp var tracon = builder.AddTracon() .UseVoice(builder.Configuration.GetSection(VoiceOptions.SectionName)) .UseVoiceConversation(options => { options.MaxConcurrentConnectionsPerTenant = 5; options.MaxConnectionDuration = TimeSpan.FromMinutes(30); options.IdleTimeout = TimeSpan.FromMinutes(2); options.MaxUtteranceDuration = TimeSpan.FromSeconds(60); options.MaxUtteranceBytes = 8 * 1024 * 1024; options.PersistAudio = false; }); var app = builder.Build(); app.MapTracon("/tracon"); ``` `MapTracon` maps `/api/voice/sessions/{sessionId}/stream` and installs WebSocket middleware only when the conversation services exist. Without `UseVoiceConversation()`, the route does not exist. ### Protocol The negotiated subprotocol is `tracon.voice.v1`. The client sends JSON control frames and binary audio: 1. Send `{"type":"start","agent":"voice-assistant","inputFormat":"webm-opus"}`. 2. Send audio chunks as binary frames. 3. Send `{"type":"commit"}` when client-side voice activity detection finds the end of speech. 4. Read `transcript`, `runStarted`, `text`, `audioStart`, binary audio, `audioEnd`, and `done` frames. 5. Read `idle` — a commit that closed no turn. Handle it or your client hangs. 6. Send `cancel` for barge-in or `stop` to close cleanly. :::caution[A commit does not always produce a turn] Two ordinary cases end a commit without a turn: the commit reaches the server before any audio does (voice activity detection fired on a cough, or the user pressed send inside the recorder's first timeslice), and the audio transcribes to nothing usable (a noisy room). Both answer with a single `idle` frame and no `done`. Your client must treat `idle` as "the turn is over, start listening again" — return to the listening state and reopen the microphone if you stopped it to commit. Do not fold it into `done`: `done` closes a turn that exists and carries that turn's number and cancelled flag, so a client that handles the two alike rewrites the record of the previous turn. A client that ignores `idle` stays in whatever pending state it entered when it committed, and never recovers. ::: Input formats are `webm-opus` and raw mono 16-bit little-endian `pcm16`. WebM Opus is the browser-friendly default. Raw PCM uses `InputSampleRate`, 16 kHz by default, when the server adds a WAV header for transcription. The server does not run voice activity detection. The 60-second utterance limit is a safety net for a client whose VAD never commits. ### WebSocket authentication Browsers cannot set an `Authorization` header on the WebSocket handshake. Tracon accepts the bearer value through a second subprotocol with the `tracon.token.` prefix and validates it with the same constant-time path as HTTP. It never accepts the credential in the query string, where browser, server, and proxy logs would capture it. Authentication is not authorization. A registered [`IRunAuthorizationHandler`](/guides/embedding/#6--run-and-session-authorization) is asked before the socket upgrades, with `SessionAccess.Voice` and the session id from the route, so a host can stop one user from speaking into another user's conversation. A denial refuses the handshake with `404` — the same answer an unreachable session already gives, so a refusal never confirms the conversation exists. A session id the server has never seen is **not** rejected: the first turn opens it. The handler is still asked, so only you decide whether that caller may open a new conversation under that id. ## Provider-hosted live voice Some providers host the whole spoken conversation themselves and carry the audio straight to the browser over WebRTC, full duplex. Tracon creates the session behind its tenancy, role, authorization, and concurrency gates, and turns each delegation from the model into an ordinary run. [Provider-hosted live voice](/guides/live-voice/) covers how to enable it, its settings, tenant policy and credentials, the endpoints, what the path gives up, and its cost. ## Privacy and retention Voice is personal data. `PersistAudio` defaults to `false`; transcripts and normal run records can still exist. When audio persistence is enabled, the `ready` frame tells the client and the console shows the recording state. Apply attachment and session retention policies, document consent, and test deletion before production. On the provider-hosted path the audio is never stored, because it never reaches Tracon — but the **text** is a separate question. `PersistTranscript` defaults to `true`, which writes the conversation into the agent session's durable history so that `GET /api/sessions/{id}` shows the full exchange and delegated runs see the whole context. That is a real retention decision, so it is handled two ways: - Set `PersistTranscript = false` to keep the transcript in memory only. Delegation keeps working with the same context for the session's lifetime; only the durable write is skipped. - Either way the effective value comes back in the session-creation response as `persistTranscript`, so a client can show it. No recording happens silently. The `MaxSpokenCharactersPerTurn` default is 5,000. Text beyond it still appears as captions but is not synthesized, which bounds voice cost without hiding the complete answer. ## Production checklist - [ ] The reverse proxy permits WebSocket upgrade and preserves the full path. - [ ] Load balancing is sticky for the lifetime of a connection. - [ ] Proxy and server idle timeouts exceed the configured conversation idle timeout. - [ ] Connection limits match file descriptors, memory, provider concurrency, and tenant quotas. - [ ] `OutputMediaType` matches the configured synthesis format. - [ ] Audio persistence, consent, access, backup, and retention policy are explicit. - [ ] Barge-in, reconnect, maximum duration, provider outage, and slow clients are tested through the real proxy. For the provider-hosted path: - [ ] The trade in [what it gives up](/guides/live-voice/#what-the-provider-hosted-path-gives-up) is acceptable for this workload, and the gap in content-guard coverage is written down. - [ ] `PersistTranscript` matches your retention policy and your users were told. - [ ] The model is priced per minute, and dashboards add the session's run costs to `cost` rather than showing it alone. - [ ] `MaxConcurrentSessionsPerTenant` and `MaxSessionDuration` bound the spend a single tenant can cause. - [ ] Load balancing is sticky: a live session binds to the instance that created it. ## Troubleshooting ### The voice WebSocket route returns `404` Call `UseVoiceConversation()` before `builder.Build()`. The route is conditional and does not exist without that registration. `MapTracon()` maps it; there is no separate voice mapping call. ### The WebSocket request returns `400` The request did not complete a WebSocket upgrade. Use a WebSocket client and the mapped route `/api/voice/sessions/{sessionId}/stream`. `MapTracon()` installs the required middleware when voice conversation is registered. ### The WebSocket request returns `401` Browsers cannot add an `Authorization` header to the handshake. Send the token in the documented `Sec-WebSocket-Protocol` subprotocol. Do not put it in the query string; URLs reach browser, server, and proxy logs. ### Startup stops with `VoiceOptions: no API key was given` The built-in ElevenLabs client serves at least one of the two speech contracts, and it has no key. The message names the contract it still serves. Set the key with `dotnet user-secrets set "Tracon:Voice:ApiKey" "..."`, or register your own `ISpeechSynthesizer` and `ISpeechTranscriber` **before** `UseVoice()`. A registration after `UseVoice()` is not seen, and the message is the same. ### A speech call returns `502` with `HTTP 403 (model_not_found)` The OpenAI project behind your key cannot use the speech model. Open the project in the OpenAI dashboard, allow the model, or set a model the project can use in `SynthesisModel` or `TranscriptionModel`. `HTTP 401` means the key itself is refused. ### A speech call returns `403` The tenant's egress policy does not name the provider the engine reports, or the tenant has its own key for that provider and the engine cannot take one. The message names the tenant and the provider. Add the provider name to the policy, or remove the binding; `GET /api/voice/providers` shows each engine's provider name and whether it accepts a tenant key. ### A dialogue node fails before any audio is made The run's error names the cause. `cannot voice a dialogue` means the synthesizer under that name does not implement `ISpeechDialogueSynthesizer`; name one that does. `The dialogue has N characters` or `uses N voices` means the request is over the limits in `DialogueCapabilities`; split it. `ProviderSettings is invalid` names the key that is unknown, belongs to another provider or has no effect on a dialogue. A provider `400` for a turn that holds only an inline tag such as `[pause]` means the turn has no speakable text: the provider refuses it. ### Voice conversation reports `501` The conversation driver exists, but transcription or synthesis does not. Register both `ISpeechTranscriber` and `ISpeechSynthesizer`. `Tracon.Voice.UseVoice()` provides the built-in ElevenLabs pair when its key and model settings are valid. ### Synthesized audio is rejected as an attachment The attachment store validates file signatures. The built-in default `mp3_44100_128` has a detectable MP3 format. Headerless PCM and µ-law output formats cannot pass the default attachment guard without a deliberate custom storage and validation design. ## In the reference - [Voice HTTP endpoints](/http-api/voice/) ## Read next - [Attachments and multimodal input](/guides/multimodal/) — attach validated files, images, and audio to a conversation. - [Observability and cost](/guides/observability/) — interpret telemetry, pricing, and recorded token use. - [Production deployment](/guides/production/) — prepare migrations, replicas, health checks, and operational settings.