Data Quality on AWS: Glue Data Quality, DataBrew and Skew Handling

Data quality is measured along named dimensions, and the exam expects you to recognize which dimension a described defect violates. The core list:

• Completeness — required values are present; null or empty customer_id fields fail it.
• Accuracy — values reflect reality; a recorded temperature of 500 degrees fails it.
• Consistency — the same fact agrees across datasets and systems; an order total that differs between the operational store and the warehouse fails it.
• Validity (conformity) — values match the expected format, type or domain; a date stored as 31/02/2025 or a state code outside the allowed set fails it.
• Uniqueness — entities appear once; duplicate primary keys fail it.
• Timeliness (freshness) — data arrives within its expected window; yesterday's partition missing at 9 a.m. fails it.
• Integrity — references resolve; an order row pointing at a nonexistent customer fails it.

These dimensions are not academic garnish — they are the vocabulary of the tooling. AWS Glue Data Quality rule types map almost one-to-one onto them (IsComplete and Completeness for completeness, IsUnique and IsPrimaryKey for uniqueness, ColumnValues for validity, DataFreshness for timeliness, ReferentialIntegrity for integrity, RowCountMatch and AggregateMatch for cross-dataset consistency), and DataBrew's profile output reports missing values, duplicates and type mismatches per column. When a question describes a symptom — "analysts report the same customer appears three times" — translate it to the dimension (uniqueness), then to the rule or check that enforces it. That two-step translation solves most quality questions on the exam.

Quality checks belong inside the pipeline, at the point of processing, because a bad record caught mid-flight costs one branch decision, while the same record discovered in the warehouse costs a backfill and a credibility conversation. The simplest and most-tested check is empty-field detection: flag records whose required columns are null or blank. In a Glue job the managed way is the Evaluate Data Quality transform with a DQDL ruleset; in a Lambda-based stream processor it is a per-record guard clause; in SQL-based flows it is a WHERE clause separating good rows from bad.

What you do with failing records matters as much as detecting them, and there are three postures. Fail fast: stop the job when quality falls below threshold — right when downstream consumers must never see bad data, such as financial reporting. Quarantine: route failing records to a separate S3 prefix or table (a dead-letter pattern for data) and let the clean majority proceed in the same run — right for high-volume feeds where a fraction of bad rows should not block the whole load, and it preserves the evidence for debugging. Tag and continue: stamp each row with a quality flag and let consumers decide — right for exploratory or ML workloads.

Row-level outcomes: how quarantine works in Glue

When the Evaluate Data Quality transform outputs the original data with row-level results enabled, it appends four columns to every row: DataQualityRulesPass, DataQualityRulesFail, DataQualityRulesSkip (arrays of rule names) and DataQualityEvaluationResult (Passed or Failed). Route on that last column — for example with a Conditional Router transform — so passed rows go to the curated target and failed rows go to a quarantine location, in the same run. Only rules that can be judged per row (IsComplete, ColumnValues and similar) flag individual rows. Dataset-level rules such as RowCount, Sum or RowCountMatch describe the whole batch, so at row level they land in DataQualityRulesSkip and can never quarantine a row; act on them through the job-level result instead.

Fail-fast gating: the On ruleset failure action

The transform's On ruleset failure setting decides what a failed ruleset does to the job:

For a gate, place the evaluation before the target write and choose fail without loading. Because the default is None, a ruleset can fail on every run while the job stays green — which is why the next section on alerting matters.

AWS Glue Data Quality is the purpose-built rules engine for the Glue ecosystem: you define a ruleset in the Data Quality Definition Language (DQDL), evaluate it, and get a per-rule pass or fail plus an overall score. It runs in two modes. Against a Data Catalog table, it checks data at rest on demand or on a schedule. Inside a Glue ETL job, the Evaluate Data Quality transform checks data in motion and can produce row-level results for quarantine, as described above.

Rules you should be able to read

Null is not the same as blank. IsComplete and Completeness count only nulls. A blank field in a CSV file read into a String column arrives as an empty string, not null, so it passes both rules. To catch blanks, use the ColumnValues keywords EMPTY and WHITESPACES_ONLY (with NULL) as shown above.

Dynamic rules compare a metric with its own history using last(k) and functions such as avg, min and max — the right tool for "alert if today's load is far smaller than usual" without hard-coding a number. They are supported in Glue ETL jobs only, not in Data Catalog rulesets. Glue Data Quality also offers analyzers with anomaly detection for the same purpose when you do not want to pick thresholds yourself.

Recommendations versus column statistics

Rule recommendations scan a table and draft a DQDL ruleset from observed statistics (this column was never null, these values were always in this range). They are a starting point you review and edit before saving — they encode what the data has been, not what it must be. Do not confuse them with Data Catalog column statistics, which compute descriptive numbers for a table (null counts, distinct values, minimum and maximum, lengths) for query planning and discovery but propose no rules.

One scoping note: the official guide names DataBrew as its example service, but Glue Data Quality with DQDL is AWS's main quality offering and appears in practice; know both, and tell them apart by workflow — DataBrew is visual and analyst-friendly, Glue Data Quality is DQDL-driven and pipeline-native.

Quality results reach people through Amazon EventBridge and Amazon CloudWatch, not through job failure. When the Publish metrics to Amazon CloudWatch option (on by default) stays selected, each evaluation emits CloudWatch metrics in the Glue Data Quality namespace — such as glue.data.quality.rules.passed and glue.data.quality.rules.failed — and an EventBridge event from source aws.glue-dataquality with detail-type Data Quality Evaluation Results Available. Clear that option and neither is emitted.

Two practical patterns: an EventBridge rule matching that detail-type with state FAILED, targeting an Amazon SNS topic; or a CloudWatch alarm on glue.data.quality.rules.failed greater than zero. Keep the two job-related signals apart:

This distinction catches many designs: a quarantining job (On ruleset failure = None) succeeds even when rules fail, so job-failure events and alarms never fire. Alert on the data quality event or metric instead. DataBrew has its own equivalent, the DataBrew Ruleset Validation Result event.

Orchestration. If AWS Step Functions runs the job and must react to a failed gate, call Glue with the startJobRun.sync (run a job) integration so the state waits for the run to finish and fails when the job fails; add a Catch that routes to an SNS notification or cleanup state. The plain request-response integration returns as soon as the run starts, so the workflow never sees the failure. A failure only exists to catch if the gate actually fails the job — that is, the On ruleset failure action is not None.

AWS Glue DataBrew approaches quality from the analyst's side: a visual, no-code service for profiling and preparing data. A profile job runs against a dataset and produces per-column statistics — missing value counts, distinct and duplicate counts, type distribution, min, max, percentiles, correlations and outliers. Profiling is the reconnaissance step of any quality effort: before you can write sensible rules, you need to know what the data actually looks like.

DataBrew also has its own data quality rules: you define a ruleset — column X has no missing values, column Y is unique, values in column Z fall within a range or match a pattern — and associate it with a dataset. Rulesets are validated only when a profile job runs, not by recipe jobs, and each validation emits a DataBrew Ruleset Validation Result event to EventBridge as well as a per-rule pass or fail report. DataBrew can also suggest rules from a profile. Downstream of assessment, DataBrew recipes fix what profiling found — over 250 built-in transformations to fill or drop missing values, deduplicate, standardize formats and split columns — applied at scale by recipe jobs without writing Spark. Recipes clean; profile jobs and rulesets measure.

The gotcha to remember: a DataBrew project (the interactive session) works on a sample — first n rows, last n rows or a random selection, up to 5,000 rows — and a profile job by default reads a custom row count (20,000 rows) unless you choose the full dataset. A session or default profile can therefore look clean while rare defects hide in the rows it never read, which is why sampling technique is on the exam.

Investigating consistency means proving that data agrees with itself across hops — source to landing zone, landing zone to warehouse, warehouse to report. Start with reconciliation counts: row counts per table and per partition at each stage, compared automatically after every load. A count mismatch localizes the loss to a hop, which is most of the diagnosis. But counts only prove the number of rows agrees: a load that changes values — truncated decimals, a shifted currency, a wrong join — keeps the count identical. Catch value drift with aggregate checksums: sums of numeric measures, hashes over sorted keys or rows, or column aggregates compared source to target per partition.

Within a dataset, consistency questions become rule checks: cross-field logic (ship date not before order date), referential integrity (every fact row's foreign key resolves to a dimension row) and cross-dataset agreement. When both datasets can be read in one Glue job, DQDL has multi-dataset rules that use a reference alias: RowCountMatch (row counts agree within a ratio), AggregateMatch (for example, the sum of a column matches between primary and reference), DatasetMatch (rows match on key columns) and ReferentialIntegrity (values exist in the reference dataset). Profiling both sides with DataBrew and comparing the reports — distributions, distinct counts, null rates — also reveals quickly where two supposedly identical datasets diverge.

Two recurring causes deserve names. At-least-once duplicates: streaming and retrying pipelines may deliver a record more than once, and append-only writes then leave the target with more rows than the source. Fix the write, not the report: make it idempotent with a merge (upsert) on a unique event id — MERGE INTO on an Apache Iceberg table, for example — and add a uniqueness rule. Timing skew: comparing a source at 9:00 against a target loaded at 8:00 manufactures phantom inconsistency — reconcile against a common watermark or partition boundary, not wall-clock now.

Uniqueness across incremental loads. An IsUnique or IsPrimaryKey rule evaluated only on each new micro-batch proves the batch has no internal duplicates; it cannot see a row that repeats one loaded yesterday. Evaluate uniqueness on the target table after the load (or deduplicate with MERGE so the target cannot hold repeats). When a scenario says "the dashboard does not match the source system," the systematic answer walks the pipeline hop by hop with counts and checksums, rather than re-running the load and hoping.

Sampling trades completeness of inspection for speed and cost, and the right technique is the one whose bias fits the goal. Random sampling selects every record with equal probability — the default for estimating overall quality metrics because it is unbiased, though small subpopulations may be missed entirely. Stratified sampling divides data into groups (by region, event type, customer tier) and samples each group — often oversampling the small ones — guaranteeing rare-but-important segments are represented. Systematic sampling takes every nth record — cheap, but biased when the data has a period that lines up with the interval. Cluster sampling picks whole groups (for example, a few complete partitions or files) and inspects everything in them — cheap to read, but only representative if the clusters resemble each other. First-n or last-n (head or tail) sampling grabs rows from one end — fastest of all and the most biased, since files are usually ordered by time or key; fine for checking schema, wrong for measuring quality.

These choices are concrete in AWS tooling. A DataBrew project samples FIRST_N, LAST_N or RANDOM rows, up to 5,000. A DataBrew profile job reads either the full dataset or a custom number of rows (20,000 by default). Athena offers TABLESAMPLE (BERNOULLI or SYSTEM) for quick approximate exploration of large tables.

The key judgment: samples estimate; they cannot certify. A random sample can estimate a ratio rule such as "about 3% of emails are missing" with known error. It cannot prove uniqueness: a duplicate is only visible if both copies land in the sample, which at a 10% sample happens for about 1 in 100 pairs — so a handful of duplicates in millions of rows will almost always be missed. The same goes for any rare defect. Sample for profiling, distribution analysis and quick investigation; run gate rules such as IsUnique, IsPrimaryKey or IsComplete against the full batch, as Glue Data Quality does inside an ETL job.

Data skew is an uneven distribution of records across partitions in a distributed job, and its signature is clear: in Spark on Glue or EMR, 199 of 200 tasks finish in seconds while one grinds for an hour, holding the whole stage up — or dies with an executor out-of-memory error. The cause is a hot key: a null-heavy join column, one customer generating half the events, or a partition key where one value dominates. Skew is also a quality concern because it usually reflects a data property (unexpected nulls, default values, runaway duplicates).

Why more partitions do not help. A shuffle sends every row with the same key to the same partition. Raising spark.sql.shuffle.partitions or repartitioning on the hot key spreads the other keys more thinly, but the hot key still lands, whole, in one task. To split it you must change the key (salting) or avoid the shuffle (broadcast).

Null keys. All null join keys hash to the same partition, so a column that is 40% null produces one giant task even though nulls never match anything. Filter them out of the join and add them back afterwards for a left join.

Aggregations differ from joins. AQE's skew handling applies to joins. Built-in aggregations such as sum and count are partially aggregated on each executor before the shuffle, so a hot key sends only one partial result per task and rarely hurts. Skew bites when per-group work cannot be combined early — for example a grouped pandas function (applyInPandas), which loads a whole group into one task's memory and can run out of memory on a hot key. Prefer built-in aggregations where they express the logic; salt with a two-stage aggregation only when per-key work cannot be partially aggregated.

Scenario to remember: a Glue join of clickstream events to sessions runs forever; the Spark UI shows one task shuffling far more data than the rest, and profiling reveals 40% of events carry a null session ID from a broken mobile client. The complete answer spans both halves of this lesson — isolate or quarantine the nulls (a completeness rule would have caught them at ingestion), then let AQE or salting handle any remaining hot keys. Fixing skew without fixing the defect that caused it treats the symptom only.