How Generative AI Models Work: Tokens, Model Choice and Deployment (AI-901)

A generative AI model turns a prompt into a completion by predicting a probable next token, appending it to the text, and repeating that step until it predicts the end of the response. The input text is the prompt; the generated text is the completion (in a chat app, the assistant's reply).

The pipeline behind every request has four stages:

1. Tokenize — the prompt is split into tokens, each mapped to an integer ID.
2. Embed — each token ID becomes a vector of numbers, combined with a positional encoding that records where the token sits.
3. Attend — transformer layers use attention to adjust each token's vector according to the tokens around it, producing contextual representations.
4. Predict — the model outputs a probability for every token in its vocabulary, one token is sampled (temperature and top_p shape this choice; at very low temperature the most probable token almost always wins), and the loop repeats.

Two consequences follow. First, the model does not look anything up: it does not search its training documents, pick a prewritten answer or run rules against a database. It continues text using statistical patterns it learned during training. Second, because it produces a plausible-sounding continuation, it can write fluent, confident text that is not true — for example a citation to a report that doesn't exist. Its knowledge is limited to its training data, and it has no built-in sense of where that knowledge ends. Applications reduce this by grounding the prompt with current, trusted data (covered with retrieval and agents in the generative AI workload lessons).

A token is the unit of text a language model reads and writes: a whole word, part of a word, a punctuation mark or another frequent character sequence, each mapped to an integer ID by the model's tokenizer. Common short words are usually one token; longer or rarer words are split into several sub-word pieces (for example a prefix such as un plus the rest of the word); punctuation marks are tokens in their own right.

Tokens matter because everything on a model is counted in them, never in words or characters:

• the context window — how much a single request can hold;
• max output tokens — how long one response can be;
• the deployment's tokens-per-minute (TPM) allocation — how much traffic it accepts;
• pricing — input and output tokens are billed (output usually at a higher rate).

That is why a cost estimate built by counting words comes out low: sub-word splits and punctuation make the token count higher than the word count.

An embedding is a vector of floating-point numbers that represents the meaning of a piece of text. Texts with similar meanings produce vectors that point in similar directions, so you can compare meaning mathematically — most often with cosine similarity (the closer to 1, the more similar).

Embeddings appear in two places:

• Inside every language model, token IDs are converted to embedding vectors before the transformer layers process them.
• As a product in their own right, an embeddings model (such as text-embedding-3-small or text-embedding-3-large) takes text and returns one vector per input. These vectors power semantic search, recommendations, clustering and the retrieval step of retrieval-augmented generation: a query for a waterproof jacket can match a product described as a rain shell even though the two share no words.

The properties of an embedding to know:

Embeddings also differ from keyword approaches: keyword extraction or keyword search still needs shared words, while embeddings compare meaning.

A transformer is the neural network architecture behind today's language models; it uses attention to work out how much each token should be influenced by every other token, and a positional encoding to keep track of word order.

• Attention. For each token, attention layers weigh the other tokens in the sequence by relevance. In the river burst its bank after the storm, the representation of bank draws more on river and storm than on the or its — which is how the model knows this is a riverside, not a financial institution. Multi-head attention runs several attention calculations in parallel, each picking up a different kind of relationship. The output is a contextual embedding: the same word gets a different vector in a different sentence.
• Positional encoding. Attention on its own treats the input as a set, not a sequence. The teacher thanked the student and the student thanked the teacher contain exactly the same tokens with the same IDs, so the model adds a positional encoding to each token's vector to mark where it sits. Order changes meaning, and positional encoding is what carries it.
• Encoder and decoder blocks. The original transformer has an encoder block that builds contextual representations of the input and a decoder block that generates output one token at a time, attending to what it has produced so far. Generative chat models (the GPT family, for example) are decoder-based; models built for understanding text, such as BERT, are encoder-based.

A context window is the maximum number of tokens a model can handle in a single request, and the input and the output share it. Everything the model sees and writes for one call — system message, conversation history, grounding data, the user's question, the generated reply and, for reasoning models, the hidden reasoning tokens — must fit inside that one budget.

Language models are stateless: a chat app keeps a conversation going by resending the history with every request. In a long conversation that history keeps growing until the request no longer fits and the call fails with an input-too-long error. The fix is to keep each request within the window — trim older turns, summarize them, or keep a sliding window of recent messages — or to choose a model with a larger context window.

Keep two different limits apart: the context window caps the size of one request, while a deployment's throughput settings (tokens per minute, provisioned throughput) cap how much traffic it accepts per minute across all requests.

You choose a model in the Microsoft Foundry model catalog, the hub where you discover, compare and deploy models from Microsoft, OpenAI and many other providers. You can filter by capability, task, provider and deployment option, and every model has a model card describing what it does, its inputs and outputs, limits and benchmark results.

To compare candidates before you deploy anything, use the model leaderboards, which rank models on quality, safety, cost and throughput, and the side-by-side comparison view. Once you have picked a model, you deploy it and can try it in the playground.

The catalog groups models into two families:

Pick the model type from the input and output the solution needs, then from its constraints (latency, cost, where it runs).

The near-twins to keep apart: a multimodal chat model reads an image and answers in text, an image-generation model goes the other way (text to picture), and an image-analysis service such as Azure AI Vision tags or captions existing pictures. Spoken input or output is a separate capability again: it needs an audio-capable model variant, such as the GPT-4o audio or realtime models, rather than a standard text-and-image chat deployment.

Reasoning models are not always the better choice. They work through a problem before answering, which raises accuracy on multi-step tasks but adds latency and generates reasoning tokens — never shown in the reply, yet billed as output tokens and counted against the context window. For simple, high-volume, real-time chat where speed and cost decide, a standard chat model such as GPT-4.1 is faster and cheaper.

Large vs small language models. Large language models (LLMs) are broad and capable but typically need data-centre-class hardware, so they are usually consumed as a cloud endpoint. Small language models such as Microsoft's Phi family are compact models designed for constrained scenarios: running on-device or offline, with limited memory, or where low cost and latency matter more than breadth. What an SLM gives up is breadth of knowledge and depth of reasoning; what it gains is lower cost, lower latency and the ability to run on modest hardware. A longer context window is not one of its advantages: a small model such as Phi-4-mini supports about 128,000 tokens, while large models such as GPT-4.1 accept far more.

When you deploy a model sold directly by Azure, the deployment type decides where prompts and responses may be processed and how capacity is billed. Location comes in three scopes:

In every scope, data at rest stays in the geography you chose for the resource; the scope governs where processing happens. Microsoft recommends Global Standard as the starting point for most workloads: pay-per-token, the lowest price, broadest region coverage, the highest default quota and early access to new models. Move away from it only for a reason — residency, reserved capacity or batch work.

The second half of a deployment type's name is the capacity model, and it combines with the location scope (Global Standard, Data Zone Provisioned, Global Batch and so on).

Combine the two axes to answer a requirement. The location scope answers where processing may happen and the capacity model answers how capacity is bought, so read each requirement against its own axis. For example, US-only processing for a large job whose results can wait means Data Zone Batch: Data Zone for the location, Batch for the asynchronous, discounted processing. Reserved capacity is only ever provided by a Provisioned type, and residency only by a Data Zone or Standard/Regional scope.

Deployment options: serverless API vs managed compute. Foundry chooses how a model can be hosted based on the model itself. Serverless API is the path for Foundry Models; it offers the standard, provisioned, batch and Developer types, billed per token or by PTU, and Microsoft runs the infrastructure. Managed compute (in preview in the current Foundry) hosts open-source, partner and custom-weight models — for example open-weight models from the Hugging Face collection — on dedicated virtual machines that Foundry manages, billed hourly per accelerator (GPU) SKU whether or not requests arrive.

Tokens-per-minute quota and throttling

Tokens per minute (TPM) is the throughput allocation you assign to a deployment from your subscription's quota for that model and region; it also implies a requests-per-minute limit. When traffic exceeds it, the deployment throttles and returns rate-limit errors (HTTP 429), even though each individual request is perfectly valid.

TPM is estimated from both the prompt size and the max output tokens setting of each request, so there are two levers when throttling hits at peak: raise the TPM allocation (up to your available quota), or make each request count for fewer tokens — shorter prompts and a lower max output tokens value. Spreading requests out and retrying with back-off also help.

Request parameters control how varied and how long each response is; they are set per request (or in the playground), separately from the deployment.

• Temperature (0 to 2) sharpens or flattens the probability distribution the next token is sampled from. Low values such as 0.2 keep output focused and consistent — good for legal, factual or extraction tasks. High values give more varied, creative text. Temperature 0 makes output close to deterministic.
• Top P (top_p, nucleus sampling) limits the choice to the smallest set of tokens whose probabilities add up to the chosen value. It also controls variety, so Microsoft recommends adjusting temperature or top_p, not both.
• Max output tokens caps how many tokens one response may generate. When a long answer reaches the cap, generation simply stops — often mid-sentence — and the response is marked incomplete instead of returning an error. It also bounds cost and TPM consumption. It is called max_output_tokens in the Responses API and max_tokens or max_completion_tokens in Chat Completions.
• Stop sequences end generation when a given string appears.
• Presence and frequency penalties discourage the model from repeating tokens it has already used.

Reasoning models take different parameters. Support varies by model, so check the model card: the o-series and GPT-5 reasoning models don't accept temperature, top_p, presence or frequency penalties, logit_bias or max_tokens. Cap length with max_completion_tokens (Chat Completions) or max_output_tokens (Responses API) — a limit that includes the hidden reasoning tokens — and tune thinking depth with reasoning_effort. Some newer reasoning models, such as the GPT-6 family, do accept temperature and top_p.