Text Analysis and Speech Apps in Microsoft Foundry (AI-901)

Use Azure Language in Foundry Tools when you need consistent, structured values from a well-defined analysis, and use a deployed general-purpose model when you need flexible, instruction-driven work that combines several tasks. Both are available from the same Microsoft Foundry resource, so the choice is about the shape of the result, not about extra infrastructure.

A model's output can be constrained (for example with structured outputs or few-shot examples), but a purpose-built analyzer returns those values without any prompt design. Azure Language also offers summarization (extractive and abstractive summaries) through the same client. Which analysis technique matches which business need (key phrases, entities, sentiment, summarization) is covered in the Task 1.3 lesson on AI workloads; this lesson is about building the app.

Install azure-ai-textanalytics, create a TextAnalyticsClient from your Foundry resource's endpoint and a credential, then call the method that matches the analysis. A Foundry resource already includes Azure Language, so you do not create a separate Language resource.

from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

client = TextAnalyticsClient(
    endpoint="https://<resource>.cognitiveservices.azure.com/",
    credential=AzureKeyCredential(key))

• Endpoint: the Foundry resource endpoint (the cognitiveservices.azure.com form), not the OpenAI /openai/v1/ path.
• Credential: for key authentication, wrap the resource key in AzureKeyCredential. For Microsoft Entra ID authentication you pass a token credential such as DefaultAzureCredential() instead, and the client obtains tokens for the signed-in identity.
• No deployment name: Language analyzers are prebuilt and called directly by method, so the client needs only the endpoint and credential. Deployment names belong to calls to a deployed generative model.

Each method does one job. PII detection is the privacy analyzer: it both lists personal details and produces a masked copy of the document. Named entity recognition classifies the people, places and organizations mentioned; entity linking ties well-known ones to a knowledge-base entry; key phrase extraction summarizes what a document is about; sentiment analysis labels opinion. An app that needs several kinds of output calls one method per analysis (or bundles them with begin_analyze_actions()).

The Language SDK takes a list of documents and returns a list of results in the same order, one per document, so result n always belongs to input n. Each list item is analyzed as a separate document with its own result.

Errors are reported per document, not raised. If the service rejects one document (an empty string, for example), that position holds a DocumentError whose is_error is True, while the other documents succeed normally. Well-written code checks is_error before reading fields:

for doc in client.analyze_sentiment(reviews):
    if doc.is_error:
        print(doc.error.code, doc.error.message)
    else:
        print(doc.sentiment, doc.confidence_scores.positive)

Know where each value lives:

• Language detection: primary_language.name is the English name (for example French); primary_language.iso6391_name is the two-letter ISO 639-1 code (for example fr), which is what you use as a key in code; primary_language.confidence_score is a 0–1 certainty, useful for flagging doubtful detections.
• PII detection: redacted_text is the whole document with each detected personal detail replaced by asterisks — the copy that is safe to store. Each item in entities describes one finding, with the detected value, its type (such as PhoneNumber) and a confidence score; because entities holds the detected values as they appear in the source, it is for inspection, while redacted_text is for storage.
• Sentiment: sentiment is the label; confidence_scores holds the positive, neutral and negative scores.
• Warnings: warnings lists non-fatal processing notices for a document (for example, text that was truncated).

Use the OpenAI Python library pointed at your Foundry resource: set base_url to https://<resource>.openai.azure.com/openai/v1/, pass the Foundry key as api_key, and name your deployment in the model argument of a Responses API call.

from openai import OpenAI

client = OpenAI(base_url="https://<resource>.openai.azure.com/openai/v1/",
                api_key=key)
response = client.responses.create(
    model="my-gpt-deployment",
    input="List the product features this customer mentions and rewrite the complaint politely: ...")
print(response.output_text)

Three endpoints are easy to confuse, and each goes with a different client:

Keep endpoints and keys in configuration (a .env file or environment variables) rather than in code. Chat clients built with the Foundry SDK (azure-ai-projects) and multi-turn conversations are taught in the Task 2.1 lesson; here the model is simply another way to analyze text.

Add the Azure Language in Foundry Tools MCP server to the agent as a tool. In the Foundry agents playground you search the tool catalog for Azure Language in Foundry Tools and connect it using your Foundry resource name; the agent can then call Azure Language analyzers through the Model Context Protocol (MCP) with no integration code of your own.

The MCP server exposes Azure Language's text analyzers as tools, including language detection, sentiment analysis, named entity recognition, key phrase extraction, summarization and PII detection and redaction, among others. Because it reaches Azure Language, it works on text the agent already has. Other kinds of input belong to other Foundry capabilities:

With the tool connected, the agent decides when to call an analyzer, the analyzer does the analysis and returns its structured result, and the model uses that result in its reply. The analysis itself (what counts as personal data, how it is masked, which sentiment label applies) is done by Azure Language, so it is the same every time the tool is called.

Keep the two addresses apart: the deployed model is the agent's reasoning engine, while the Language tool connection identifies the Foundry resource. The Foundry resource already includes Azure Language, so that one connection is all the tool needs.

Send the recording to an audio-capable chat model (for example a gpt-4o-audio-preview or gpt-4o-mini-audio-preview deployment) as base64-encoded data in an input_audio content part of the user message, and the model answers the spoken question directly. To also get a spoken answer, add modalities=["text", "audio"] and an audio argument that names the voice and output format.

audio_b64 = base64.b64encode(open("question.wav", "rb").read()).decode()

completion = client.chat.completions.create(
    model="my-audio-deployment",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "wav"},
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Answer the question in this recording."},
        {"type": "input_audio",
         "input_audio": {"data": audio_b64, "format": "wav"}}]}])

reply = completion.choices[0].message
print(reply.audio.transcript)                         # the words of the spoken answer
open("answer.wav", "wb").write(base64.b64decode(reply.audio.data))

When the reply comes back as audio, its words are carried in audio.transcript rather than in message.content. Spoken output is switched on by the pair modalities + audio; without them the model answers in text only. You can also try an audio-capable deployment without code in the Foundry chat playground by recording or uploading audio.

Distinguish audio-capable chat models from transcription models such as gpt-4o-transcribe: a transcription model produces a transcript of the audio it receives. An audio-capable chat model understands the spoken prompt and generates the answer, so one call covers what would otherwise be a three-step pipeline (recognize, generate, synthesize).

Install azure-cognitiveservices-speech, create a SpeechConfig from the Foundry resource key and endpoint, then pass it — together with an audio configuration — to a SpeechRecognizer (speech to text) or a SpeechSynthesizer (text to speech).

import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(
    subscription=os.environ.get("SPEECH_KEY"),
    endpoint=os.environ.get("ENDPOINT"))

SpeechConfig is the only object that carries the key and endpoint. It also holds service settings for each direction, and the two sets are easy to mix up:

Where the audio comes from or goes to is a separate object, and the direction is in the class name:

The same filename= argument means opposite things in the two classes: AudioConfig reads a file, AudioOutputConfig writes one. A recognizer created with no audio configuration listens on the default microphone. set_speech_synthesis_output_format(...) on SpeechConfig chooses the encoding of synthesized audio (for example a RIFF/WAV format), while AudioOutputConfig chooses its destination.

Call recognize_once_async() for a single utterance and start_continuous_recognition() for long or live audio; then check result.reason to learn whether speech was recognized, nothing was heard, or the request failed.

Continuous recognition is event-driven: you connect handlers, start it, and results arrive through those handlers:

• recognizing fires repeatedly with partial, growing hypotheses — good for showing live text, wrong for building a transcript (you would repeat words).
• recognized fires once per finalized phrase — append evt.result.text here.
• session_stopped and canceled signal that recognition ended; wait for them, then call stop_continuous_recognition(), so the program does not exit before the whole file is processed.

Speech results report their outcome through result.reason (a ResultReason value).

Whenever result.text is empty, result.reason and its details explain why, so a robust app logs the reason alongside each transcript. Recognition quality also depends on speech_recognition_language matching the audio's locale.

Create a SpeechSynthesizer from the SpeechConfig and an AudioOutputConfig, then call speak_text_async() for plain text or speak_ssml_async() for SSML that controls how the text is spoken. A successful call returns ResultReason.SynthesizingAudioCompleted.

speech_config.speech_synthesis_voice_name = "en-US-AvaMultilingualNeural"
audio_out = speechsdk.audio.AudioOutputConfig(filename="reminder.wav")
synth = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=audio_out)
result = synth.speak_text_async("Your appointment is tomorrow at nine.").get()

SSML (Speech Synthesis Markup Language) is how you change delivery while keeping the same voice: a voice element names the voice, prosody adjusts rate, pitch and volume, and break inserts a pause of a set length. SSML must go to speak_ssml_async(); passing markup to speak_text_async() treats it as plain text to read.

Use the Speech SDK when you need only one direction (transcribe or speak) with fine control, an audio-capable multimodal model when a short spoken question needs an answer in one call, and Azure Speech Voice Live when you are building a real-time, two-way voice agent. Voice Live is a fully managed speech-to-speech API that combines speech recognition, a generative model of your choice and speech synthesis in one service, and handles conversational behaviour such as callers interrupting. You can use it from code with the azure-ai-voicelive package, or switch on Voice mode for an agent in the Foundry portal.

Each design trades control for simplicity: the separate pipeline exposes every step, while Voice Live manages the conversational loop for you. Azure Language works on text, so in a voice app it applies to transcripts rather than to audio.