AWS Glue Data Catalog: Crawlers, Partitions, and the Hive Metastore

A data catalog exists so that compute engines can query data where it already lives, because files in S3 are just bytes until something records their schema, format, and location. The catalog holds that metadata — table definitions, column names and types, serialization format, partition layout, and the S3 path or JDBC source behind each table — and every engine that trusts the catalog can immediately query the data at its source, with no copies and no per-engine schema definitions.

That is the pattern DEA-C01 calls "use data catalogs to consume data from the data's source." Register a dataset once, and Amazon Athena queries it serverlessly, Redshift Spectrum joins it against warehouse tables through an external schema, Amazon EMR reads it from Spark or Hive, and AWS Glue jobs load it as dynamic frames — all from the same table definition. Change the schema in one place and every consumer sees the change.

Without a catalog, each tool needs its own hand-maintained DDL, schemas drift apart, and analysts can't discover what data exists. With one, the catalog becomes the single source of truth for structure, the enforcement point for governance (AWS Lake Formation grants its fine-grained permissions on catalog databases, tables, and columns), and the discovery surface for the whole lake.

Keep the two catalog categories straight from the start: a technical catalog (Glue Data Catalog, Apache Hive metastore) stores machine-facing metadata that query engines require, while a business catalog (Amazon SageMaker Catalog) layers human-facing context — descriptions, glossaries, ownership, approval workflows — on top. The exam tests both and the boundary between them.

The AWS Glue Data Catalog is a fully managed, serverless technical catalog: one catalog per AWS account per Region, organized as databases (namespaces) containing tables, where each table records the schema, input/output formats, SerDe (serializer/deserializer), physical location, and partitions of a dataset.

A table in the catalog does not hold data — it points at data. For S3-backed tables the location is a prefix such as an S3 path per partition; for JDBC-backed tables it references a connection and a source table. Tables get into the catalog three ways: crawlers infer them automatically (next section), DDL creates them explicitly (CREATE EXTERNAL TABLE in Athena or Spark SQL), and the Glue APIs or infrastructure-as-code define them programmatically. Each table keeps a version history as its schema evolves, so you can inspect and roll back schema changes.

Referencing the catalog is what makes it useful. Athena resolves every query against it. Redshift Spectrum external schemas map to catalog databases. Glue ETL jobs read tables as dynamic frames by database and table name instead of hardcoding paths and schemas. EMR clusters can be configured to use the Glue Data Catalog as their metastore for Spark and Hive. The Data Catalog also serves as the metastore for Apache Iceberg tables on AWS, tracking each table's current metadata pointer.

Access control is two-layered: IAM policies gate the Glue APIs themselves, and AWS Lake Formation adds database-, table-, column-, row-, and cell-level grants enforced by Athena, Spectrum, Glue, and EMR. Cross-account catalog sharing (via Lake Formation grants or catalog resource policies) lets a producer account expose tables to consumer accounts without duplicating data or metadata.

The Glue Data Catalog is Apache Hive metastore-compatible: it implements the metastore interface that Hive, Spark, Presto, and Trino were built against, which is why those engines can treat it as a drop-in metastore. Understanding the relationship is an exam skill because migration scenarios hinge on it.

A traditional Apache Hive metastore is self-managed: a metastore service backed by a relational database (often MySQL or PostgreSQL) that you provision, patch, back up, and scale yourself — commonly running alongside an on-premises Hadoop cluster or on an EMR cluster's master node backed by RDS. The Glue Data Catalog delivers the same table metadata as a serverless managed service: no database to run, native integration with Athena, Spectrum, Lake Formation, and Glue ETL, and IAM-based security.

Amazon EMR is where the choice is explicit: a cluster can use its default local metastore (lost when the cluster terminates), an external Hive metastore on RDS (persistent, self-managed), or the Glue Data Catalog (persistent, managed, shared with every other AWS analytics engine). For persistent metadata with the least operational overhead — and any scenario where Athena or Redshift Spectrum must see the same tables as EMR — the Glue Data Catalog is the exam answer. The external RDS metastore remains valid when a scenario demands Hive-metastore-specific behavior or an unmigrated legacy dependency.

For migrations, the standard pattern moves Hive metastore contents into the Glue Data Catalog (AWS provides migration tooling and scripts for this), after which on-premises engines' workloads pointed at AWS use one shared, serverless catalog. When a question contrasts "self-managed metastore database" with "fully managed, serverless catalog integrated with Athena," it is drawing exactly this line.

A Glue crawler discovers schemas so you don't hand-author them: point it at a data store, and it samples the data, infers format and schema through classifiers (built-in classifiers recognize JSON, CSV, Parquet, Avro, ORC, and more; custom classifiers using grok patterns, XML, or JSON paths handle proprietary formats), groups similar objects into tables, detects the partition layout from the S3 folder structure, and writes or updates table definitions in the Data Catalog.

Crawlers read more than S3: JDBC sources (through a Glue connection), DynamoDB tables, and other stores can be crawled, which is how you catalog an operational database's schema for downstream ETL. Each crawler runs with an IAM role that must be able to read the target data and write to the catalog.

Configuration is where exam questions live. Scheduling: run on demand, on a cron schedule, or triggered by events — including event-driven crawls fed by S3 event notifications through SQS so the crawler only examines new objects instead of re-listing the whole prefix. Incremental behavior: crawling new folders only makes repeat runs cheap on append-only datasets. Schema-change policy: you choose whether the crawler updates the table definition, logs the change without altering the table (the safe choice when downstream jobs depend on a stable schema), and whether deleted objects mark tables deprecated or delete them. Table grouping: a single top-level folder with consistently structured subfolders becomes one partitioned table, while inconsistent structures splinter into many tables — a common misconfiguration symptom the exam describes as "the crawler created hundreds of tables instead of one." The fix is pointing the crawler at the right folder level or instructing it to create a single schema for compatible prefixes.

Partitions must be registered in the catalog before queries can see them: Athena and Spectrum prune by partition using catalog metadata, so a new dt=2026-07-29 folder in S3 is invisible until something adds the matching partition entry. Synchronizing partitions is its own exam skill with four mechanisms to rank.

Re-run a crawler — the general answer. A scheduled or event-driven crawler detects new partition folders and adds them, handling schema drift at the same time. Cost: crawler runtime and scheduling lag.

MSCK REPAIR TABLE in Athena scans the table's S3 path and adds any Hive-style partitions (key=value folder names) missing from the catalog. Simple, but it lists the entire prefix — slow and list-heavy on tables with many partitions, and it only works for Hive-style layouts.

ALTER TABLE ADD PARTITION registers specific partitions explicitly — the precise, cheap move when your pipeline knows exactly which partition it just wrote. A common production pattern has the ETL job or a Lambda add the partition immediately after writing, eliminating both crawler lag and full-prefix scans. (Glue ETL jobs can also update the catalog with new partitions as they write.)

Partition projection removes synchronization entirely: you declare partition patterns (date ranges, integer ranges, enums) in table properties, and Athena computes partition locations at query time instead of consulting stored partition metadata. It is the answer for highly partitioned, predictable layouts where partition management itself became the bottleneck.

For tables keeping millions of catalog partitions, partition indexes speed up partition lookup during query planning.

Scenario

A pipeline writes hourly Parquet files to s3://lake/events/dt=YYYY-MM-DD/hr=HH/, and analysts complain new hours don't appear in Athena until the nightly crawler runs. Lowest-latency, lowest-cost fixes: have the writer issue ALTER TABLE ADD PARTITION after each write, or switch the table to partition projection over the date and hour ranges — either beats scheduling the crawler hourly.

A Glue connection is a catalog object that stores how to reach a data store — connection type, endpoint, credentials, and network placement — so crawlers and ETL jobs can use sources and targets that aren't public S3 paths.

JDBC connections cover relational sources and targets: Amazon RDS, Aurora, Amazon Redshift, and on-premises or self-managed databases reachable over JDBC. The connection stores the JDBC URL and credentials — best practice (and the exam answer) is referencing a secret in AWS Secrets Manager rather than embedding a username and password. Purpose-built connection types also exist for DynamoDB, MongoDB, Kafka/MSK, and a growing set of SaaS sources.

Networking is the part candidates get wrong. To reach a database inside a VPC, the connection specifies a VPC, subnet, and security group; when a job or crawler runs, Glue provisions elastic network interfaces (ENIs) in that subnet, and traffic originates from them. That drives three classic troubleshooting facts: the database's security group must allow inbound from the ENIs' security group (a self-referencing rule on a shared security group is the common setup); the subnet needs a route to whatever else the job touches — an S3 gateway endpoint (or NAT gateway) so the same job can read the database and write to S3; and DNS resolution must be enabled in the VPC for hostname-based JDBC URLs. If a scenario says a Glue job in a VPC connects to the database but fails writing to S3, the missing S3 endpoint or NAT route is the answer.

Connections are reused across the system: a crawler uses one to catalog a source database's schema, and ETL jobs use the same connection as a source or a target — for example writing transformed output back to Redshift or Aurora. Test-connection functionality validates reachability and credentials before you debug a failing job blind.

A business data catalog serves people rather than query engines: it makes data assets discoverable, understandable, and governed for the whole organization. On AWS that is Amazon SageMaker Catalog, part of the next generation of Amazon SageMaker (built on Amazon DataZone technology) and surfaced through SageMaker Unified Studio.

In SageMaker Catalog, technical assets — Glue Data Catalog tables, Redshift tables, and other data and ML assets — are enriched with business metadata: plain-language descriptions, business glossary terms, ownership, and quality context, with generative-AI assistance for drafting descriptions. Work is organized into domains and projects; producers publish assets to the catalog, and consumers search for them and subscribe, triggering an approval workflow. When the owner approves, access is granted automatically through the underlying permission machinery (Lake Formation grants for lake tables, Redshift grants for warehouse tables) — governed self-service instead of tickets and manual bucket policies.

They are complements, not rivals: the business catalog references assets whose schemas live in the technical catalog. Exam cue: schema-for-queries requirements point to the Glue Data Catalog; discovery, business context, and governed subscription workflows point to SageMaker Catalog.