AIP-C01 Data Validation and Processing Pipelines for Foundation Models

A data validation and processing pipeline for foundation model (FM) consumption is the set of steps that checks, converts, formats and enriches data before a model reads it, so that every request is valid and every answer is built from clean, unambiguous input. A model cannot tell a truncated article, a mislabelled number or a misheard drug name from the truth; it simply uses what it is given. Fixing data upstream is cheaper and more reliable than asking the model to compensate.

Task 1.3 covers four jobs, and each has its own AWS tools:

Neighbouring tasks own the next steps: chunking, embeddings and retrieval belong to Tasks 1.4 and 1.5, prompt templates and prompt governance to Task 1.6, and PII detection and redaction to the safety and privacy material in Domain 3. A recurring rule runs through this whole topic: use deterministic code or a purpose-built service for anything that must be exact, and use a model only for language work.

Validate batch datasets with AWS Glue Data Quality: you declare rules in DQDL (Data Quality Definition Language) and Glue evaluates them, with no validation code to write or maintain. Rules run in two places: inside a Glue ETL job through the Evaluate Data Quality transform (the check sits in the same job that writes the data a knowledge base syncs from), or on a schedule directly against Glue Data Catalog tables. Glue can also generate recommended rules from a table's statistics, which is a starting ruleset to review and enforce, not a profiling report.

Pick each rule's form to match the requirement's tolerance:

Two details catch people out. First, with threshold is the share of rows that must pass, so a 2% tolerance is > 0.98, never > 0.02; it applies to ColumnValues, ColumnDataType and CustomSQL. Second, a threshold only decides whether the rule as a whole passes; it never removes or routes individual records. Checking presence (!= NULL) is not the same as checking allowed values.

Act on Glue Data Quality results with one of four levers: row-level outcomes, the fail-job option, CloudWatch rule metrics or EventBridge score events; catch drift with dynamic rules or DetectAnomalies. The right lever depends on whether you need to handle individual records, stop a run, or alert a person:

The evaluation event carries run-level facts (score, rules passed and failed), never the failing records, and it fires after the run, so it cannot route records. Glue publishes these EventBridge events only while the Publish metrics to Amazon CloudWatch option is selected on the evaluation (it is selected by default); clearing it to save cost silently stops the events. AWS service events arrive on the default event bus with no bus policy, and a CloudTrail-based rule matches API calls, a different event shape.

Validating against history instead of constants

Data that feeds a model drifts: volume grows, an upstream bug truncates text. Fixed thresholds need constant re-tuning and miss partial damage, so compare each run with its own history:

• Dynamic rules state a documented, deterministic formula: RowCount > avg(last(5)) * 0.75 or Mean "token_count" > avg(last(14)) * 0.6. last(k) with k greater than 1 returns a list, so it must be reduced with an aggregate such as avg, min or max; comparing with a bare last(5) is invalid. Choose the aggregate that matches the requirement (average is not minimum).
• DetectAnomalies (DetectAnomalies "RowCount", DetectAnomalies "Completeness" "summary") records each run's value and, once it has enough history, fails when a new value is anomalous by a learned model. Use it when you want no thresholds at all; use a dynamic rule when auditors need an explicit formula.
• Analyzers gather statistics (and feed anomaly detection) but cannot assert anything or fail a run.

Dynamic rules and analyzers are supported only in AWS Glue ETL, not in Data Catalog evaluations. To use them on a catalog table, evaluate the table inside an ETL job with the Evaluate Data Quality transform.

Use Glue Data Quality for datasets, a Lambda function for single requests or single files, and SageMaker Data Wrangler for interactive exploration. Glue evaluates tables and job outputs; it cannot sit in a synchronous API path or check one uploaded file as it arrives.

Making custom validation observable

Publish rejections as CloudWatch custom metrics. The embedded metric format turns a structured log line from Lambda into a metric, and a reason dimension lets one dashboard show rejections by cause. For a rate alarm (for example, more than 5% of an hour's files fail), publish both the valid and the invalid counts and divide them with metric math; an alarm needs a numerator and a denominator. Do not use the Lambda Errors metric for this: a function that detects an invalid file and quarantines it completes successfully, so Errors counts code faults and retries, not invalid data. Counting Bedrock ValidationException errors only measures failures after the fact and rejects nothing in advance. S3 Lifecycle rules (age, prefix, tags) and Glue crawlers (schema inference) do not validate content.

Data Wrangler for exploration

SageMaker Data Wrangler is now part of SageMaker Canvas. Its data quality and insights report flags missing values, duplicate rows, outliers and class imbalance, and its data flows apply cleanup transforms visually on the same data. It is an exploration tool, not a production gate. To run a finished flow on every full-size export as a repeatable, versioned step, export the flow to SageMaker Pipelines rather than re-implementing the steps by hand in Lambda or rewriting them as DQDL (which checks data but cannot transform it).

Convert each modality into something the model reads well, using the managed service built for it. The three most-tested building blocks are Amazon Transcribe, Amazon Bedrock Data Automation and SageMaker Processing; direct image and document input to Bedrock models is covered with request formatting below.

Audio: Amazon Transcribe

Do not mix separate channels down to mono; that discards the signal that made attribution reliable. Fixing misheard terms in a later prompt only corrects the summary, not stored transcripts that other systems (such as a search index) read. To start the next step without polling, react to the Transcribe Job State Change event in EventBridge (status COMPLETED or FAILED) instead of looping on GetTranscriptionJob.

Mixed documents, images and audio: Bedrock Data Automation

Amazon Bedrock Data Automation (BDA) extracts structured output from documents, images, audio and video through one managed API, so you do not build and orchestrate a separate Textract, Rekognition and Transcribe pipeline per file type. It has two output modes:

• Standard output: generic insights with nothing to define, such as extracted text, layout elements and summaries for documents, or transcripts for audio.
• Custom output with blueprints: you define the fields to extract (policy number, incident date, damage description), configured in a BDA project. Use blueprints only when you need your own fields; parsing business fields out of generic summaries is the custom code blueprints remove.

The synchronous InvokeDataAutomation operation is meant for single images; process documents, audio and video with InvokeDataAutomationAsync. Instead of polling GetDataAutomationStatus, set the request's notification configuration with EventBridge enabled and start downstream work from the completion event.

Large-scale preprocessing: SageMaker Processing

A SageMaker Processing job runs your own container (for example OCR or native libraries) on managed, temporary instances: it copies input from S3, runs, uploads output to S3 and releases the instances. It suits hours-long preprocessing that Lambda cannot do, since a Lambda invocation stops at 15 minutes, and it is not a hosting option like a real-time endpoint. Glue Python shell jobs run Python scripts, not your container image.

• To scale out, set the S3 input's distribution type to ShardedByS3Key, which gives each instance a different subset of objects. FullyReplicated copies every object to every instance, so each one redoes the whole corpus.
• Only files written under the output's local path (for example /opt/ml/processing/output) are uploaded. Files written elsewhere, such as /tmp, are discarded, and the job still reports Completed. EndOfJob versus Continuous upload mode changes when that path is uploaded, not which path.

Prepare tabular data by doing the arithmetic in the database and sending the model a small, labelled result. Models are poor calculators and every row costs input tokens, so pasting a raw export into a prompt overflows the context window and still produces wrong totals.

• Aggregate at the source. Run the sum, count or group-by in Amazon Athena, a database or a query and pass only the result. Vector search is no substitute: top-k similarity returns a few matching rows, not all the rows a total needs.
• Label every value. Serialize rows as JSON objects or Markdown tables with column names and units (unit_cost_eur, reorder_level), never bare positional values such as 8812,7.50,9.25,40. Wrapping header-less values in a CSV document block leaves them just as ambiguous.
• Filter rows and project columns. Query only the period a question concerns and only the attributes the answer needs. This is the largest input-token saving, because a document block's contents are still input tokens.

Three adjacent settings are often misapplied here. maxTokens caps the response length; it does not enlarge the context window for input. Splitting rows across several messages does not help either, because the whole conversation must still fit in one context window. Prompt caching saves cost only when later requests reuse an identical prompt prefix; with one question per session there is nothing to reuse. Lowering temperature makes output more repeatable, but it cannot tell the model which unlabelled number is the price.

Bedrock offers two request styles: InvokeModel, whose JSON body is specific to each model provider, and the Converse API, which uses one request and response shape across supported models. Use Converse when code must stay model-agnostic, so that switching models means changing only the model ID; use InvokeModel when you need a model's native body. The contentType header only says the body is JSON; it never translates one provider's format into another's.

Other providers have their own InvokeModel bodies (Amazon Titan Text uses inputText and textGenerationConfig; Meta Llama uses prompt and max_gen_len), which is exactly why Converse exists. inferenceConfig has no topK member; additionalModelResponseFieldPaths selects extra fields to return, and settings written into the system prompt are just text, not parameters.

Images and documents in Converse

• Image blocks accept png, jpeg, gif and webp, each up to 3.75 MB and 8,000 pixels per side, and up to 20 images per message. Convert unsupported formats (such as TIFF or BMP) and downscale oversized photos and split large sets across requests.
• Document blocks accept pdf, csv, doc, docx, xls, xlsx, html, txt and md, up to five documents of up to 4.5 MB each. PowerPoint is not a document format, so convert decks to PDF; rendering a long deck as one image per slide breaks the image cap and turns text into pixels. The format must describe the real file.
• Images and documents are accepted only in user-role messages, not in assistant messages or the system field.
• A message that carries a document must also carry a text block in the same message.
• A document name may contain only letters and numbers, single spaces, hyphens, parentheses and square brackets, so a file name such as q3_report.v2.docx is rejected. AWS advises a neutral name ("Document A"), because the name is a prompt-injection surface.
• Presigned URLs pasted as text are just text; the model does not fetch them.

Batch inference input

Bedrock batch inference (CreateModelInvocationJob) is asynchronous processing of many prompts from S3, not interactive chat. Its input is JSON Lines in files ending in .jsonl, one object per line (a JSON array in a .json file fails validation), and each record has exactly two parts: recordId and modelInput. The model is chosen once on the job, never per record. modelInput follows the job's model invocation type: the model's native InvokeModel body by default, or a Converse request body when the job's invocation type is Converse, which lets you reuse existing Converse bodies unchanged. AWS does not guarantee that output order matches input order, so set recordId to your business key (Bedrock generates one if you omit it) and join results on it, never on line position or on IDs the model writes into its text.

A SageMaker AI endpoint receives the request body exactly as sent, so the serving container defines the payload schema and the ContentType tells it how to parse the bytes. Nothing translates Bedrock formats. For a container that uses the Hugging Face Text Generation Inference (TGI) schema, as many SageMaker JumpStart text models do, the prompt goes in inputs and generation settings in a parameters object:

{"inputs": "<prompt>",
 "parameters": {"max_new_tokens": 512, "top_p": 0.9}}

Titan, Llama or Converse bodies are valid for Bedrock, which makes them tempting distractors, but this container cannot read them. Large or slow requests also change how you invoke the model:

Asynchronous endpoints also solve idle cost and completion signalling. Register the variant with Application Auto Scaling with MinCapacity 0 and a target-tracking policy on ApproximateBacklogSizePerInstance (optionally a step policy on HasBacklogWithoutCapacity to scale out from zero quickly); requests that arrive at zero instances queue until capacity returns. Configure success and error SNS topics in the endpoint's async notification settings so downstream services are told when results are ready, instead of a scheduled function listing the output prefix. Deleting and recreating an endpoint between bursts drops requests instead of queueing them.

Format a conversation as an ordered list of role-tagged messages, oldest first, ending with the new user message, and re-send that history on every call. The Converse API is stateless: a model has no memory between calls, so an application that sends only the latest message gets a bot that forgets. Keeping user and assistant turns as separate messages preserves who said what; flattening the transcript into one block of text in the system field loses that structure. Instructions that define the assistant belong in system, not in a fake first message.

Long sessions eventually exceed the context window, and cost and latency grow with every turn, so history must be bounded without losing facts the user stated early:

Prompt caching reuses the processing of a repeated prefix to save cost and latency; it does not store a conversation or raise the window limit. maxTokens limits only the reply length. Durable storage of session history (for example in DynamoDB) and clarification workflows are covered with interactive systems in Task 1.6.

Enhance input by normalizing it with deterministic code, rewriting free-form language with a small model, and adding structured facts extracted by a purpose-built service. The dividing line is simple: if the result must be exactly the same every time, use code; if the task is language, use an FM.

Deterministic normalization in Lambda

• Convert dates, units and number formats with rule-based code. A model is probabilistic, so it cannot guarantee identical conversions that auditors require.
• Normalize text before hashing, deduplicating or embedding: apply one Unicode normalization form such as NFC (so a base letter plus a combining accent becomes the precomposed character), and collapse each run of whitespace, including non-breaking spaces, to a single blank and trim the ends. Stripping non-ASCII characters destroys accented text, and an embedding-similarity threshold is a different, lossy kind of deduplication.

Reformatting with a Bedrock model

A small, inexpensive Bedrock model is well suited to expanding shorthand ("rplcd fltr, tstd pmp") into clear sentences or turning free text into a record, before the main model runs. When that record must match a schema every time, supply the JSON schema as structured output in the Converse request's outputConfig.textFormat (type json_schema, with the schema under structure.jsonSchema) on a model that supports it. Stronger prompt wording and temperature 0 only make stray prose less likely, retry loops add code without a guarantee, and guardrails enforce content policies, not formats.

Entity enrichment with Amazon Comprehend

A custom recognizer analyses single documents in real time only through a Comprehend endpoint, billed for its provisioned inference units over time. Asynchronous analysis jobs are batch work with start-up delay, far too slow for a synchronous request path. For a predictable daily pattern, scale the endpoint's inference units with Application Auto Scaling (scheduled scaling) instead of paying for peak capacity around the clock.