AI-103: Speech Solutions — Voice Agents, Custom Speech and Speech Translation

Azure Speech in Foundry Tools offers four speech-to-text paths, and the right one follows from where the audio is and how fast the text is needed: real-time recognition with the Speech SDK for live audio, the fast transcription API for one file at a time with a synchronous answer, batch transcription for large sets of stored files, and LLM Speech (the enhanced mode of fast transcription) when a language model should shape the output.

Two output forms matter. The display form has punctuation, capitalization and formatted numbers ("$25.50"); the lexical form is the raw recognized words ("twenty five dollars fifty"). Downstream tools that need lexical text rule out fast transcription. Cost and hosting also decide: a custom speech model must be deployed to a paid custom endpoint for real-time recognition and speech translation, while batch transcription references the model directly. Fast transcription and LLM Speech have no custom-model option at all: they run on base models, optionally helped by a phrase list. Batch transcription's language identification works only with base models; to pair language detection with a custom model, use real-time recognition (see the custom speech section).

For agents, real-time is the usual path for the conversation itself, while fast or batch transcription handles recordings the agent must analyse afterwards.

Text to speech in the Speech SDK is a SpeechSynthesizer built from a SpeechConfig: you pick the voice with speech_synthesis_voice_name (for example en-US-AvaMultilingualNeural), then call speak_text_async for plain text or speak_ssml_async for SSML markup. Each config property has one job, and the near-twins are easy to mix up:

SSML controls delivery. Wrap text in a prosody element with a rate (and pitch or volume) to change pace, insert a break with a time such as 500ms for a pause at an exact point, and use mstts:express-as to apply a speaking style (cheerful, calm and so on) where the voice supports it. A style changes tone, not timing. SSML only works through speak_ssml_async; sending markup to speak_text_async makes the voice read the tags as text.

Output format must match the destination. set_speech_synthesis_output_format chooses the encoding. Riff* formats are WAV files with a header; Raw* formats have no header, which streaming and telephony gateways often require. Narrowband telephony runs at 8 kHz with G.711 companding, which comes in two variants: mu-law (North America and Japan, Raw8Khz8BitMonoMULaw) and A-law (Europe and most other regions, Raw8Khz8BitMonoALaw). Lowering the sample rate changes audio quality, not speaking speed.

Voices come as standard neural voices, higher-fidelity HD voices, and multilingual voices that can speak several languages; all are prebuilt and shared by every customer, so none of them is a unique brand voice.

Pronunciation of output speech is fixed on the text-to-speech side, with SSML elements for one-off cases and a custom lexicon for rules that should apply everywhere. Recognition-side tools (phrase lists, custom speech pronunciation data) only change what the service hears, so they never fix a voice that mispronounces a word.

A lexicon is an XML Pronunciation Lexicon Specification file, not SSML. Each lexeme holds one or more grapheme entries and either a phoneme (in an alphabet such as IPA) or an alias. An alias suits acronyms: a lexeme for SLA with the alias "service level agreement" makes every SLA spoken as the phrase. Matching is case-sensitive, so a lexeme for "Contoso" doesn't cover "CONTOSO"; add a grapheme or lexeme for each casing that appears. A lexicon covers one locale, is hosted at a URL (for example Azure Blob Storage) and is cached by the service, so edits take a short while to appear.

You reference the lexicon from SSML with a lexicon element. In the Voice Live API, the voice configuration accepts custom_lexicon_url, which applies the lexicon to every reply for standard and custom voices without editing prompts or SSML.

The Voice Live API is a managed speech-to-speech interface: in one WebSocket session it combines speech recognition, a generative model and text to speech, so an agent can hold a spoken conversation without you orchestrating three services. It adds the conversational audio features a hand-built chain lacks:

• Noise suppression for busy environments and echo cancellation so the agent doesn't hear (and answer) its own voice on a speakerphone.
• Interruption detection and advanced end-of-turn detection, so the agent doesn't cut in when a caller pauses mid-sentence.
• Avatars for a talking visual front end, plus the choice of standard, HD, custom and personal voices.
• Natively supported models that are fully managed (nothing to deploy), or your own model deployment.

You configure the session with a session.update event: instructions, the voice object, input_audio_transcription (the recognition model, such as azure-speech or an OpenAI transcription model like gpt-4o-transcribe or whisper-1, plus phrase_list and custom_speech), and turn-detection, noise and echo settings.

Agent mode connects Voice Live to an existing Foundry Agent Service agent instead of a bare model. The connection names the agent (agent_name) and its Foundry project (project_name), optionally a version or a conversation to resume, and the agent brings its own instructions, model and tools, so nothing is copied into the session. Agent invocation requires Microsoft Entra ID authentication; API keys aren't supported for it.

A phrase list is the fast, no-training fix for a modest set of misrecognized words; a custom speech model is the trained fix for large vocabularies, acoustic conditions, or workloads where phrase lists don't apply.

The "where it doesn't" row explains a common symptom: a phrase list that changes nothing. In a batch job, tuning the list can't help; the answer is a custom model trained on text that uses the terms. In Voice Live, switching input_audio_transcription to azure-speech lets the phrase list take effect. Putting names in an agent's system prompt doesn't help either, because the model sees the words only after they've already been transcribed.

A custom speech model is a base model adapted with your data; you choose the data type by the kind of error, measure it with word error rate (WER), and then make the app actually call it.

Choosing training data

Text data is cheap to collect and targets vocabulary; audio data is the only way to improve acoustic robustness. Pronunciation data here affects recognition, not how a voice speaks.

Testing with WER

WER compares what a model recognizes from audio with a human-labeled transcript of the same audio, so an accuracy test needs audio paired with human-labeled transcripts. Audio alone supports only listening to results, and text data can't score anything. Measure the base model first; if its WER is acceptable, you may not need a custom model at all, and the same test set compares later models.

Using the model

• Real-time recognition and speech translation (Speech SDK): deploy the model to a custom endpoint, then set endpoint_id on the SpeechConfig (or SpeechTranslationConfig) to that endpoint's ID. Without endpoint_id, the service uses the base model for the locale, however good the trained model is.
• Fast transcription and LLM Speech: there is no custom-model or custom-endpoint option; they use base models, and a phrase list is the runtime fix for vocabulary.
• Batch transcription: reference the model in the job; no endpoint is needed.
• Language detection with a custom model: build AutoDetectSourceLanguageConfig from SourceLanguageConfig objects, each with a locale and optionally a custom endpoint ID. The custom model then applies only when its language is detected, and one recognizer handles the call. A global endpoint_id would force the custom model on every caller.
• Voice Live: set custom_speech under input_audio_transcription to a dictionary keyed by locale (for example zh-CN) whose values are custom model IDs; other locales stay on the base model. It works only with the azure-speech transcription model, and the model must be available on the same Foundry resource that calls Voice Live, so a model trained on another resource has to be copied over first.

A unique brand voice comes from professional custom voice; a voice that copies an individual end user comes from personal voice; everything else is a prebuilt standard or HD voice.

Personal voice fits scenarios such as voice banking for patients who will lose their speech, or speaking translated output in each speaker's own voice. A custom speech model is not a voice at all: it improves recognition. In Voice Live, the custom voice deployment must be on the resource the session calls, and only the voice type, name and (for a custom voice) endpoint_id decide which voice speaks; other voice settings change how that voice speaks, not who it is.

Multimodal reasoning from audio means sending the audio itself to an audio-capable model rather than a transcript, so the model can use tone, emotion and non-speech sounds (sirens, alarms, laughter) that any text conversion throws away. Sentiment on a transcript, diarization or a transcription prompt still work on words only.

Audio in chat completions

Audio-capable chat models such as gpt-4o-audio-preview and gpt-4o-mini-audio-preview accept an audio content part next to text in a user message:

{"type": "input_audio",
 "input_audio": {"data": b64_audio, "format": "wav"}}

The data is base64-encoded audio, not a file path or URL, and it is not an image_url part or base64 pasted into text. modalities=["text"] asks for a text reply; for a spoken reply as well, set modalities=["text", "audio"] and pass audio={"voice": "alloy", "format": "wav"}. This is request/response: good for analysing a recording, not for a live conversation.

The GPT Realtime API

For live, low-latency speech with a speech-native model, the GPT Realtime API streams audio both ways with a gpt-realtime deployment. The transport follows from the client:

A browser must never hold a long-lived key or Entra ID token. The standard WebRTC pattern: your backend, using its own key or Entra ID credential, mints a short-lived ephemeral token from /openai/v1/realtime/client_secrets; the browser uses it to post its SDP offer to /openai/v1/realtime/calls. For stricter rules, the backend can also proxy the SDP negotiation, so no credential of any kind reaches the browser. The negotiation returns a Location header for the call, which the backend can use to open an observer WebSocket to record or control the session's events.

Speech translation in the Speech SDK recognizes speech in one language and returns text (and optionally audio) in one or more others, using a SpeechTranslationConfig and a TranslationRecognizer.

• The source is a language-locale string: speech_recognition_language = "ja-JP".
• Each target is a language code: add_target_language("en"), add_target_language("es"). One recognition pass then fills result.translations, keyed by those codes.
• Spoken output through voice_name and the synthesizing event works only for a single target language, and the voice should match that language.

To speak several languages from one recognition pass, add every target, then send each text translation to a SpeechSynthesizer with a voice in that language. Running one recognizer per language recognizes the same audio repeatedly.

Live Interpreter, part of real-time speech translation, handles conversations where nobody can choose the language in advance: it continuously identifies the spoken language with open-range detection (no candidate list; in Python an empty AutoDetectSourceLanguageConfig), follows speakers who switch language mid-session without restarting, and can speak the translation in the speaker's own personal voice (limited access). Standard translation with continuous language identification still needs candidate locales chosen up front.

Language models translate speech in three ways, and the choice depends on the target language and whether text must be reviewed in between.

The two-step pattern wins whenever the source transcript is a durable artifact: compliance can approve it, and when a glossary changes you re-run only the text translation. Direct audio-to-text translation (LLM Speech, the Speech SDK translation recognizer, whisper) reprocesses the audio every time and leaves no approved transcript in between.

Text analysis of the resulting transcript (entities, summaries, sentiment, structured output) belongs to the language-model text analysis topic of this domain.