Data Quality on AWS: Glue Data Quality, DataBrew and Skew Handling

Data quality is measured along named dimensions, and the exam expects you to recognize which dimension a described defect violates. The core list:

• Completeness — required values are present; null or empty customer_id fields fail it.
• Accuracy — values reflect reality; a recorded temperature of 500 degrees fails it.
• Consistency — the same fact agrees across datasets and systems; an order total that differs between the operational store and the warehouse fails it.
• Validity (conformity) — values match the expected format, type or domain; a date stored as 31/02/2025 or a state code outside the allowed set fails it.
• Uniqueness — entities appear once; duplicate primary keys fail it.
• Timeliness (freshness) — data arrives within its expected window; yesterday's partition missing at 9 a.m. fails it.
• Integrity — references resolve; an order row pointing at a nonexistent customer fails it.

These dimensions are not academic garnish — they are the vocabulary of the tooling. AWS Glue Data Quality rule types map almost one-to-one onto them (IsComplete for completeness, IsUnique for uniqueness, DataFreshness for timeliness, referential rules for integrity), and DataBrew's profile output reports missing values, duplicates and type mismatches per column. When a question describes a symptom — "analysts report the same customer appears three times" — translate it to the dimension (uniqueness), then to the rule or check that enforces it. That two-step translation solves most quality questions on the exam.

Quality checks belong inside the pipeline, at the point of processing, because a bad record caught mid-flight costs one branch decision, while the same record discovered in the warehouse costs a backfill and a credibility conversation. The simplest and most-tested check is empty-field detection: filter or flag records whose required columns are null or blank. In a Glue Spark job that is a filter on the DataFrame; in a Lambda-based stream processor it is a per-record guard clause; in SQL-based flows it is a WHERE clause or a conditional expression separating good rows from bad.

What you do with failing records matters as much as detecting them, and the exam rewards knowing the three postures. Fail fast: stop the job when quality falls below threshold — right when downstream consumers must never see bad data, such as financial reporting. Quarantine: route failing records to a separate S3 prefix or table (a dead-letter pattern for data) and let the clean majority proceed — right for high-volume feeds where a fraction of bad rows should not block the whole load, and it preserves the evidence for debugging. Tag and continue: stamp each row with a quality flag and let consumers decide — right for exploratory or ML workloads that can weight or exclude flagged rows.

Whichever posture you choose, emit the outcome as telemetry: publish counts of failed rows as CloudWatch metrics, and alert through Amazon SNS or an EventBridge rule when failure rates cross a threshold. A quality check nobody hears about is a check that silently rots. This wiring is exactly where task 3.4 overlaps the monitoring skills of task 3.3 — quality results are pipeline signals like any other.

AWS Glue Data Quality is the purpose-built rules engine for the Glue ecosystem: you define a ruleset in the Data Quality Definition Language (DQDL), evaluate it, and get a per-rule pass or fail plus an overall score. It runs in two modes. Against a Data Catalog table, it checks data at rest — schedule evaluations to continuously score a dataset. Inside a Glue ETL job, the Evaluate Data Quality transform checks data in motion as it flows through the job, and can route passing and failing rows to different outputs — the managed version of the quarantine pattern, no hand-rolled filters required.

DQDL rules are declarative and readable, and the exam may show you one. Representative examples: IsComplete "customer_id" (no nulls), ColumnValues "age" between 0 and 120 (validity), IsUnique "order_id" (uniqueness), RowCount > 1000 (volume sanity), Completeness "email" > 0.95 (at least 95 percent populated — thresholds let you tolerate known noise), and DataFreshness rules for timeliness. Glue Data Quality can also recommend rules by scanning a dataset and proposing a starter ruleset from observed statistics, which you then prune — recommendations encode what the data has been, not what it must be, so review them before trusting them.

Results integrate with the operations stack: evaluation outcomes surface as metrics and events, so an EventBridge rule can catch a failed evaluation and notify an SNS topic or trigger remediation. One scoping note for the exam: the official guide names DataBrew as the data-quality example service, but Glue Data Quality with DQDL is AWS's flagship quality offering and appears in practice; know both, and distinguish them by workflow — DataBrew is visual and analyst-friendly, Glue Data Quality is DQDL-driven and pipeline-native.

AWS Glue DataBrew approaches quality from the analyst's side: a visual, no-code service for profiling and preparing data. A profile job runs against a dataset and produces per-column statistics — missing value counts, distinct and duplicate counts, type distribution, min, max, percentiles, correlations and outliers. Profiling is the reconnaissance step of any quality effort: before you can write sensible rules, you need to know what the data actually looks like, and a profile report answers that in one run.

DataBrew also has its own data quality rules: you define a ruleset — column X has no missing values, column Y is unique, values in column Z fall within a range or match a pattern — associate it with a dataset, and the rules are validated when a profile job runs, producing a pass or fail validation report per rule. That distinction matters for exam answers: DataBrew validates quality through profile jobs, whereas Glue Data Quality evaluates rulesets against catalog tables or inside ETL jobs. Downstream of assessment, DataBrew recipes fix what profiling found — over 250 built-in transformations to fill or drop missing values, deduplicate, standardize formats and split columns — applied at scale by recipe jobs without writing Spark.

Choose DataBrew when the actors are analysts or data stewards who need visual, interactive quality work; choose Glue Data Quality when quality gates must run headlessly inside production pipelines with rule-based routing. The gotcha to remember: DataBrew works interactively on a sample of the data in its session view (the full dataset is processed only when a job runs), so an interactive session can look clean while rare defects hide in the unsampled majority — which is precisely why sampling technique, covered below, is on the exam.

Investigating consistency means proving that data agrees with itself across hops — source to landing zone, landing zone to warehouse, warehouse to report. Start with reconciliation counts: row counts per table and per partition at each stage, compared automatically after every load. A count mismatch localizes the loss to a hop, which is most of the diagnosis. Strengthen counts with aggregate checksums — sums of numeric measures, hashes over sorted keys, or column-level aggregates compared source-to-target — which catch corruption and truncation that identical row counts hide.

Within a dataset, consistency questions become rule checks: cross-field logic (ship date not before order date), referential integrity (every fact row's foreign key resolves to a dimension row), and cross-dataset agreement (the daily revenue in the summary table equals the sum over the detail table). Glue Data Quality expresses many of these directly in DQDL, and profiling both sides with DataBrew and comparing the reports — distributions, distinct counts, null rates — quickly reveals where two supposedly identical datasets diverge.

Two recurring exam-flavored causes deserve names. Ingestion-time divergence: streaming pipelines with at-least-once delivery produce duplicates that inflate one side of a comparison — deduplicate on a business key before reconciling. Timing skew: comparing a source at 9:00 against a target loaded at 8:00 manufactures phantom inconsistency — reconcile against a common watermark or partition boundary, not wall-clock now. When a scenario says "the numbers in the dashboard do not match the source system," the systematic answer walks the pipeline hop by hop with counts and checksums, rather than re-running the load and hoping.

Sampling trades completeness of inspection for speed and cost, and the exam wants you to pick the technique whose bias profile fits the goal. Random sampling selects every record with equal probability — the default for estimating overall quality metrics because it is unbiased, though small subpopulations may be missed entirely. Stratified sampling divides data into groups (by region, event type, customer tier) and samples each group, guaranteeing rare-but-important segments are represented — the right answer when a question stresses that minority classes must appear in the sample. Systematic sampling takes every nth record — cheap and simple, but dangerous when the data has periodicity that aligns with the interval, which silently biases the sample. First-n (head) sampling grabs the first rows — fastest of all, and the most biased, since files are often ordered by time or key; fine for eyeballing schema, wrong for measuring quality.

These choices are concrete in AWS tooling. DataBrew lets you configure how interactive sessions and profile jobs sample — first-n or random, with configurable size — and profiles can also run on the full dataset when accuracy justifies the cost. Athena offers TABLESAMPLE for quick approximate exploration of large tables without full scans.

The judgment the exam probes: sampled checks are for estimating quality cheaply; certain guarantees require full evaluation. A rule like "order_id is unique" verified on a 10 percent sample proves nothing about duplicates in the other 90 percent. So sample for profiling, distribution analysis and quick investigation; run gate-keeping rules that protect downstream consumers against the full data, as Glue Data Quality does inside an ETL job.

Data skew is an uneven distribution of records across partitions in a distributed job, and its signature is unmistakable: in Spark on Glue or EMR, 199 of 200 tasks finish in seconds while one grinds for an hour, holding the whole stage hostage — or dies with an executor out-of-memory error. The cause is a hot key: a null-heavy join column, one customer generating half the events, or a partition key like country where one value dominates. Skew is a quality concern because it usually reflects a data property (unexpected nulls, degenerate defaults, runaway duplicates) and because it breaks the pipelines that would otherwise catch such problems.

Know the mitigation toolkit and when each applies. Salting: append a random suffix to the hot key (splitting NULL into NULL_0 through NULL_9), spreading one giant partition across many tasks, and replicate the matching side of a join accordingly — the classic fix for skewed join keys. Broadcast joins: when one side of the join is small, broadcast it to every executor so no shuffle of the large, skewed side happens at all — often the cleanest answer when a question mentions a small dimension table. Repartitioning: explicitly repartition on a higher-cardinality or composite key so work spreads evenly. Adaptive Query Execution (AQE): modern Spark can detect skewed shuffle partitions at runtime and split them automatically — the low-effort first resort on current Glue and EMR versions. Isolate or pre-filter hot keys: handle null or degenerate keys in a separate cheap path instead of joining them at all.

Scenario to internalize: a Glue join of clickstream events to sessions runs forever; the Spark UI shows one task shuffling far more data than the rest, and profiling reveals 40 percent of events carry a null session ID from a broken mobile client. The complete answer spans both halves of this lesson — filter or quarantine the nulls (a completeness rule would have caught them at ingestion), then salt or pre-filter the key so residual imbalance cannot stall the join. Fixing skew without fixing the quality defect that caused it treats the symptom only.