Databricks GenAI Engineer: Monitoring, Inference Tables and LLM Cost Control

The right metrics follow from what can go wrong for the people who depend on the deployment: an interactive chat user cares about how quickly text appears, a nightly batch job cares about finishing before its deadline, finance cares about tokens and spend, and everyone cares about errors and answer quality. Pick the metric that isolates the stated risk rather than the one that is easiest to collect.

Two distinctions matter most. TTFT versus total latency: with streaming, users start reading as soon as the first token arrives, so a complaint about an initial wait is measured by TTFT, not by how long the full answer takes. Per-request latency versus throughput: when no user waits on any single request, as in a weekend job summarising an archive of support tickets, per-request latency is irrelevant and throughput compared with the remaining work tells you whether the deadline is at risk. Request volume, index sync lag and similar signals are real but answer different questions; a quality concern needs a quality metric and a cost concern needs a token or DBU metric.

An inference table is an AI Gateway feature that logs the traffic of a Model Serving endpoint into a governed Unity Catalog Delta table you can query with SQL: the raw request and response payloads together with the status code, latency, the requester, the served entity that answered (served_entity_id) and the Databricks request ID. You enable it on the endpoint; the application code does not change. It is the feature to use whenever someone needs the actual text of prompts and replies, for audit, debugging or building evaluation data.

The table is filled by a background delivery process, not inline with each request, and that shapes how you use it:

• Delay: delivery is best effort, and rows typically appear within about an hour. An empty table minutes after a load test usually just means the rows have not landed yet. For anything you must see right now, use real-time MLflow traces instead.
• Oversized payloads: the table stores payloads only up to a maximum size (the limit differs between the legacy endpoint gateway and the newer Unity Gateway). Above it, the request or response column is null and the logging_error_codes column says why, for example MAX_REQUEST_SIZE_EXCEEDED or MAX_RESPONSE_SIZE_EXCEEDED. The call itself can still have succeeded with a 200.
• Error responses: logging is not guaranteed for requests that fail with errors such as 401, 403, 429 or 500. Counting throttled requests from the payload table therefore undercounts them; client logs and endpoint metrics are the better source for error volume.
• Streaming: streamed responses are aggregated into one logged response rather than dropped.

Older per-agent request logs and assessment logs are deprecated; for agents, the AI Gateway inference table and MLflow traces are the current records.

When one endpoint serves more than one entity, the served_entity_id column records which entity actually answered, so you analyse per model by grouping on it (and joining to system.serving.served_entities for names and types).

• Traffic splits. With a canary split that sends a small share of traffic to a new fine-tuned model, endpoint-level charts mix both models. Grouping the inference table by served_entity_id and comparing latency and status codes gives a per-model comparison while clients keep calling the single endpoint. The general rule: compare entities within the same time window on the same endpoint, so traffic mix and time of day affect both equally and the client contract stays unchanged.
• Fallbacks. AI Gateway fallbacks (for external models) send a request on to the next served entity when one returns 429 or a 5XX error, trying each entity once. The inference and usage tables log the first successful attempt or, if every entity fails, the last failed attempt. A provider outage covered by a fallback therefore shows up as rows served by the fallback entity, not as failed rows, and users see nothing wrong.

Usage tracking writes one row per request to the system table system.serving.endpoint_usage with exact input and output token counts, the requester, the status code, the served entity ID and the request ID, but no prompt or response text. It is the consumption and chargeback record, and the right choice when a privacy rule forbids storing payloads.

• Descriptive attributes such as endpoint name, entity type (foundation, external or custom model) and task live in system.serving.served_entities. Join the two on served_entity_id.
• Exact tokens plus text: to read the prompts behind the costliest requests, join endpoint_usage to the inference table on the Databricks request ID. Neither table alone has both exact token counts and payloads; response length is not a token count.
• requester is the identity whose permissions were used for the call. Behind an application backend that calls with one service principal, every row shows that principal.
• usage_context fixes that for reporting on an endpoint with AI Gateway usage tracking: a string-to-string map the caller sends as a field of the JSON request body (with the OpenAI Python client, which has no named parameter for it, inside extra_body) that is copied into the usage_context column. Putting an end-user or customer ID there lets you group token counts per user or tenant without creating identities or endpoints. The documentation introduces it with external model endpoints and notes it for custom model endpoints too; the rule to remember is that it travels in the body, under that exact key.

You assess a deployed RAG application by turning what production logged into evaluation data and scoring it with judges whose inputs the logs actually contain. Inference-table request and response payloads can be parsed into a DataFrame of inputs and outputs and passed to mlflow.genai.evaluate without a predict function, so last month's real answers get scored as they were given.

The judge you can use depends on what was recorded:

So an inference table that holds only the final question and answer supports answer-level judges such as RelevanceToQuery, while questions about retrieval need traces with retriever spans, and correctness needs ground truth. Usage and billing data measure cost and errors, never quality.

The same logs serve a second purpose: regression testing on real traffic. Before deploying a rebuilt version (new chunking, a new prompt), pull last month's real user questions from the inference table into an evaluation dataset and run mlflow.genai.evaluate on both the live and the candidate version with the same scorers. Synthetic questions are invented, and production monitoring can only see a version after it is live. Choosing the scorers themselves is covered in the lesson on evaluating LLMs and agents.

Deploying an agent with databricks.agents.deploy() creates the serving endpoint with real-time MLflow tracing to the active experiment, AI Gateway inference tables and the Review App, with no further configuration. Several things it does not do: it does not choose or start production monitoring scorers, set rate limits, or enable scale to zero (that is an opt-in parameter). Those are separate decisions.

Traces versus inference tables for live debugging. Traces arrive in the experiment in near real time and each one contains the agent's spans; a RETRIEVER span records exactly which documents were returned. So when an agent starts answering badly shortly after a release, its traces show what the retriever actually returned for those questions, while inference table rows may not have been delivered yet. Health signals for upstream components, such as a vector index's sync state, can explain a cause but do not show what a given request retrieved.

Endpoints created another way. If you create the serving endpoint yourself through the REST API or SDK rather than with agents.deploy(), set the environment variable ENABLE_MLFLOW_TRACING=true on the served entity so the instrumented agent emits traces in production. Tracing is configured where the agent runs: settings made in a development notebook stay in that session, and the gateway's logging features (usage tracking, inference tables) record requests at the endpoint rather than instrumenting the agent's internal steps.

Agent (production) monitoring runs registered scorers automatically on a sample of the live traces in the agent's MLflow experiment and attaches the results to each trace as feedback, so quality trends appear without you building or scheduling a job. Results land after an initial processing delay; the documentation advises allowing roughly 15 to 20 minutes before the first scores appear.

• Register, then start. A scorer (built-in judges such as Safety, Guidelines or RetrievalRelevance, or a custom @scorer) is registered to the experiment and started with a ScorerSamplingConfig: a sample_rate (1.0 scores every trace) and an optional filter_string to choose which traces are eligible. Each scorer has its own configuration.
• Change or pause: update() changes the sampling of a running scorer; stop() sets its sample rate to zero. Both affect traces from now on.
• Score the past with a backfill. Starting a scorer covers new traffic only. To assess traces already in the experiment, call backfill_scorers from databricks.agents.scorers. Without a time window it covers the last seven days; start_time and end_time set the window explicitly, for example 30 days back.
• Custom scorer rules. Monitoring runs scorers outside your session, so a custom @scorer must be defined in a Databricks notebook with its imports inside the function. Module-level imports that work in an interactive mlflow.genai.evaluate run are missing at monitoring time, and class-based scorers are not supported. In general, a scorer that fails in monitoring but works interactively points at how it was packaged, not at how traces are selected.

Compare the alternatives: a scheduled job running mlflow.genai.evaluate on inference table rows works but means building a job and waiting on table delivery; a data profiling monitor watches statistical drift in a table, not judged answer quality; guardrails block content in the request path but produce no quality scores.

AI Gateway rate limits enforce a ceiling: once a caller exceeds it, further requests are rejected with HTTP 429. That is the difference from inference tables, usage tracking and budgets, which only observe. When the requirement is to stop a runaway agent loop or one heavy caller, you need a limit; when it is to see or report usage, you need a table or an alert.

Limits are measured in QPM (queries per minute) or TPM (tokens per minute) and are set at several scopes:

Precedence. A custom limit (user, service principal or group) overrides the User (Default) limit for that identity. When a user has both a user-specific limit and a group limit, the user-specific limit is enforced. A user who belongs to several groups is limited only after exceeding all of their group limits of the same type, which in practice means the most generous one applies: each group limit is evaluated on its own, not combined into a shared pool. The endpoint limit still caps total traffic for everyone. The number of limits (and group limits) per endpoint is capped, so many-group designs should be checked against the current documented maximum.

Agents and custom models take QPM only. TPM limits are not available on custom model or agent endpoints. To cap an agent's token use, put a TPM limit on the foundation or external model endpoint the agent calls; if nothing else calls that endpoint, its endpoint-level TPM limit is exactly the agent's total. Per-employee call limits go on the agent endpoint as per-user QPM.

Limits key on identity, not on usage_context. usage_context attributes usage in reports but no rate limit reads it. To both report and cap usage per tenant on one shared endpoint, give each tenant its own service principal with a custom limit and report by requester.

LLM serving costs fall when you stop paying for idle capacity, match the billing model to the traffic shape, batch offline work, and generate fewer tokens. Diagnose which of those is the problem first, then pick the lever.

Reading the scenario is the skill. Reserved capacity that is mostly idle, for a model that is also offered pay-per-token, is a billing-model mismatch: the fix is the billing model, because levers that act on requests (limits, chargeback) leave the reserved-capacity charge unchanged. Work that already lives in a table and has no user waiting belongs in ai_query batch inference rather than a client loop of real-time calls. When usage data shows replies far longer than anyone reads, capping max_tokens and asking for brevity cuts the bill directly. And idle cost on a hosted endpoint is removed by scale to zero; any always-on size, however small, still bills while idle.

Model Serving spend in DBUs is recorded in the billing system table system.billing.usage under serving SKUs, and you slice it by endpoint, by tag, or watch it with a budget.

• Per endpoint: every serving row carries usage_metadata.endpoint_name, so grouping by it and summing DBUs covers all endpoints, tagged or not.
• Per team or business unit: custom tags set on a serving endpoint (or applied through a serverless usage policy) propagate to the custom_tags column, so serving spend joins the same cost-allocation reports finance runs for jobs and warehouses. Tags on Unity Catalog models or MLflow experiments do not reach billing rows, and usage_context lives in the usage-tracking table, not in billing.
• Scale-to-zero restarts: DBUs accrued while an endpoint starts again after scaling to zero appear under a launch variant of the serving SKU (its name contains SERVERLESS_REAL_TIME_INFERENCE_LAUNCH). Seeing these rows on a scale-to-zero endpoint is expected.
• Alerts without blocking: an account budget filtered on a tag emails when spend passes a threshold. It monitors and notifies; it never rejects or slows a request. Use rate limits only when you actually want enforcement.

Choose the record by the unit asked for: DBUs and money come from system.billing.usage; tokens per caller from system.serving.endpoint_usage; text from the inference table.