AWS Glue Data Catalog: Crawlers, Partitions, and the Hive Metastore

A data catalog exists so that compute engines can query data where it already lives, because files in S3 are just bytes until something records their schema, format, and location. The catalog holds that metadata — table definitions, column names and types, serialization format, partition layout, and the S3 path or JDBC source behind each table — and every engine that trusts the catalog can immediately query the data at its source, with no copies and no per-engine schema definitions.

That is the pattern DEA-C01 calls "use data catalogs to consume data from the data's source." Register a dataset once, and Amazon Athena queries it serverlessly, Redshift Spectrum joins it against warehouse tables through an external schema, Amazon EMR reads it from Spark or Hive, and AWS Glue jobs load it as dynamic frames — all from the same table definition. Change the schema in one place and every consumer sees the change.

Without a catalog, each tool needs its own hand-maintained DDL, schemas drift apart, and analysts can't discover what data exists. With one, the catalog becomes the single source of truth for structure, the enforcement point for governance (AWS Lake Formation grants its fine-grained permissions on catalog databases, tables, and columns), and the discovery surface for the whole lake.

Keep the two catalog categories straight from the start: a technical catalog (Glue Data Catalog, Apache Hive metastore) stores machine-facing metadata that query engines require, while a business catalog (Amazon SageMaker Catalog) layers human-facing context — descriptions, glossaries, ownership, approval workflows — on top. The exam tests both and the boundary between them.

One boundary matters when the source is not S3. Athena reads Glue Data Catalog tables as files in Amazon S3, so a table that a crawler created from a DynamoDB table (useful to Glue ETL jobs, which read it through the DynamoDB connector) is not something Athena can scan. To query DynamoDB — or RDS, Redshift, CloudWatch Logs and other stores — in place from Athena, deploy an Athena Federated Query connector (a Lambda function) and register it as an Athena data source; one SQL statement can then join awsdatacatalog tables over S3 with tables in the federated catalog, reading the live source at query time.

The AWS Glue Data Catalog is a fully managed, serverless technical catalog: one Data Catalog per AWS account per Region (which can now contain nested catalogs, such as Amazon S3 Tables or Redshift-managed catalogs), organized as databases (namespaces) containing tables, where each table records the schema, input/output formats, SerDe (serializer/deserializer), physical location, and partitions of a dataset.

A table in the catalog does not hold data — it points at data. For S3-backed tables the location is a prefix such as an S3 path per partition; for JDBC-backed tables it references a connection and a source table. Tables get into the catalog three ways: crawlers infer them automatically (next section), DDL creates them explicitly (CREATE EXTERNAL TABLE in Athena or Spark SQL), and the Glue APIs or infrastructure-as-code define them programmatically. Each table keeps a version history as its schema evolves, so you can inspect and roll back schema changes.

Referencing the catalog is what makes it useful. Athena resolves every query against it. Redshift Spectrum external schemas map to catalog databases (details in the next-but-one section). Glue ETL jobs read tables as dynamic frames by database and table name instead of hardcoding paths and schemas. EMR clusters can be configured to use the Glue Data Catalog as their metastore for Spark and Hive. The Data Catalog also serves as the metastore for Apache Iceberg tables on AWS, tracking each table's current metadata pointer.

Access control is two-layered: IAM policies gate the Glue APIs themselves, and AWS Lake Formation adds database-, table-, column-, row-, and cell-level grants enforced by Athena, Spectrum, Glue, and EMR. Cross-account catalog sharing (via Lake Formation grants or catalog resource policies) lets a producer account expose tables to consumer accounts without duplicating data or metadata.

The Glue Data Catalog is Apache Hive metastore-compatible: it implements the metastore interface that Hive, Spark, Presto, and Trino were built against, which is why those engines can treat it as a drop-in metastore. Understanding the relationship is an exam skill because migration scenarios hinge on it.

A traditional Apache Hive metastore is self-managed: a metastore service backed by a relational database (often MySQL or PostgreSQL) that you provision, patch, back up, and scale yourself — commonly running alongside an on-premises Hadoop cluster or on an EMR cluster's master node backed by RDS. The Glue Data Catalog delivers the same table metadata as a serverless managed service: no database to run, native integration with Athena, Spectrum, Lake Formation, and Glue ETL, and IAM-based security.

Amazon EMR is where the choice is explicit: a cluster can use its default local metastore (lost when the cluster terminates), an external Hive metastore on RDS (persistent, self-managed), or the Glue Data Catalog (persistent, managed, shared with every other AWS analytics engine). For persistent metadata with the least operational overhead — and any scenario where Athena or Redshift Spectrum must see the same tables as EMR — the Glue Data Catalog is the exam answer. The external RDS metastore remains valid when a scenario demands Hive-metastore-specific behavior or an unmigrated legacy dependency.

Using the Glue Data Catalog as the EMR metastore has documented limits. Hive transactions (ACID tables) are not supported, and Hive authorization (SQL-standard GRANT/REVOKE) is not supported — access is controlled with IAM policies, Glue resource policies, and Lake Formation instead. A workload that depends on either is a reason to keep a self-managed metastore or to move to a table format such as Apache Iceberg for transactional writes.

For migrations, the standard pattern moves Hive metastore contents into the Glue Data Catalog (AWS provides migration tooling and scripts for this), after which on-premises engines' workloads pointed at AWS use one shared, serverless catalog. When a question contrasts "self-managed metastore database" with "fully managed, serverless catalog integrated with Athena," it is drawing exactly this line.

Each engine is pointed at a technical catalog with its own mechanism, and most configuration questions come down to knowing which setting lives where.

Amazon Redshift (Spectrum)

Redshift references lake tables through an external schema, so the tables are not redefined or loaded. CREATE EXTERNAL SCHEMA lake_ext FROM DATA CATALOG DATABASE 'lake' IAM_ROLE '...' maps a Glue database; CREATE EXTERNAL SCHEMA ... FROM HIVE METASTORE DATABASE '...' URI '...' PORT 9083 IAM_ROLE '...' maps a database in an Apache Hive metastore (for example on an EMR primary node) using the metastore's URI and port. With FROM DATA CATALOG, the IAM role named in the statement does two separate jobs: it reads metadata from the Glue Data Catalog (Glue permissions) and reads the data files from S3 (with FROM HIVE METASTORE the metadata comes from the metastore instead, but the role still needs the S3 reads) (s3:GetObject/s3:ListBucket, plus kms:Decrypt for SSE-KMS objects). If only the catalog half is granted, listing external tables and their columns works but every scan fails with an S3 access-denied error.

Redshift data sharing can include data lake tables, but only after the producer has defined them through an external schema; a datashare shares what a warehouse already references. It is therefore not a way to bring a Glue database into Redshift in the first place — the external schema is.

Amazon Athena

Athena uses the account's Glue Data Catalog (AwsDataCatalog) by default. When an existing Apache Hive metastore must stay the single source of table definitions, Athena can query through it instead: you register an external Hive metastore data source, which Athena reaches through a Lambda function running in the metastore's VPC. Queries then name that catalog, and no table is redefined in Glue.

Amazon EMR

EMR applications read their metastore client settings from configuration classifications, and Hive and Spark are configured separately:

To use a Glue Data Catalog in another account, two things are needed. In the EMR configuration, set hive.metastore.glue.catalogid to the owning account ID (in hive-site and spark-hive-site). In the catalog-owning account, add a Glue Data Catalog resource policy that grants the EMR roles from the consumer accounts access to the catalog, databases and tables; the consumer accounts also need IAM permissions for those Glue actions on the remote catalog ARNs. (If Lake Formation governs the catalog, its cross-account grants apply as well.) A resource policy placed in the consumer account grants nothing on a catalog it does not own.

A Glue crawler discovers schemas so you don't hand-author them: point it at a data store, and it samples the data, infers format and schema through classifiers, groups similar objects into tables, detects the partition layout from the S3 folder structure, and writes or updates table definitions in the Data Catalog.

Classifiers. Built-in classifiers recognize JSON, CSV (and other delimited text), Parquet, Avro, ORC, XML, common log formats and more; the first classifier that recognizes the data with enough certainty wins. When nothing recognizes a proprietary format, the table comes out classified as UNKNOWN — the fix is a custom classifier (grok patterns for line-oriented text, XML row tags, JSON paths, or a custom CSV classifier) attached to the crawler, which reads the data where it is.

The built-in CSV classifier decides whether the first row is a header with a type heuristic: it treats the first row as a header only when it differs in type from the rows below (for example, strings over numeric columns). If every column holds text, the header looks like data, so the table gets columns col0, col1, ... and the header line is returned as a row. A custom CSV classifier fixes this through its column-headings setting: Has headings declares that the first row is a header, while Detect headings just repeats the same heuristic. The custom classifier also sets the delimiter and quote symbol.

Targets and permissions. Crawlers read S3, JDBC sources (through a Glue connection), DynamoDB tables, and other stores — crawling a DynamoDB table is how a Glue ETL job gets a catalog definition for an undocumented attribute set. Each crawler runs with an IAM role that must read the target data and write to the catalog. The AWSGlueServiceRole managed policy covers Glue itself, not your data: add S3 read on the paths, and for objects encrypted with SSE-KMS under a customer managed key add kms:Decrypt on that key — without it the crawler runs, logs access-denied errors, and creates no tables.

Include paths and table grouping. A crawler compares the schemas it finds under each include path. Folders under one include path whose schemas are similar are grouped as partitions of one table named after the path, with a partition column (such as partition_0) holding the folder names. That explains both classic surprises. Two different datasets with near-identical columns under one include path (for example s3://warehouse-raw/invoices/ and s3://warehouse-raw/credit-notes/ under s3://warehouse-raw/) collapse into one table — give each dataset root its own include path to get separate tables. Conversely, one dataset whose files differ slightly (for example, newer files carry extra optional columns) can splinter into hundreds of tables — turn on Create a single schema for each S3 path (the CombineCompatibleSchemas grouping policy) so compatible schemas merge into one partitioned table, or use the table-level setting to state at which folder depth tables start.

A crawler's run mode decides what it looks at on each run, and its schema change policy decides what it is allowed to change in the catalog. Both are common sources of "the crawler ran but the catalog is wrong".

Run modes

Runs can be started on demand, on a cron schedule, or by a Glue workflow trigger. For a very large bucket where few objects change between runs, event mode is the design that avoids re-listing everything. Incremental mode suits append-only data with a stable schema; if a column is added later, it never reaches the catalog — run one full crawl with update behavior set to update the table (then switch back), or add the column yourself with ALTER TABLE ADD COLUMNS.

Schema change policy

To protect a curated table while the crawler keeps registering partitions, set update behavior to LOG (usually with partitions inheriting from the table), or use Add new columns only when new columns should still flow in but existing types must not change.

Partitions must be registered in the catalog before queries can see them: Athena, Redshift Spectrum and EMR prune by partition using catalog metadata, so a new dt=2026-07-29 folder in S3 is invisible until something adds the matching partition entry. There are four ways to register partitions, plus projection (next section), which avoids registration in Athena.

Re-run a crawler — the general answer. A scheduled or event-driven crawler adds new partition folders and, with the right update behavior, new columns too. Cost: crawler runtime and lag between the write and the next run.

MSCK REPAIR TABLE in Athena scans the table's S3 path and adds any Hive-style partitions (key=value folder names) missing from the catalog. Simple, but it lists the entire prefix — slow on tables with many partitions — and it only recognizes Hive-style layouts.

ALTER TABLE ADD PARTITION registers specific partitions explicitly — the precise, cheap move when your pipeline knows exactly which partition it just wrote. Without a LOCATION clause, Athena assumes the Hive-style default path (table location plus key=value/ folders). For non-Hive layouts such as s3://telemetry-logs/2026/03/14/, give each partition its location: ALTER TABLE telemetry ADD PARTITION (year='2026', month='03', day='14') LOCATION 's3://telemetry-logs/2026/03/14/'.

Let the Glue job update the catalog as it writes. A Glue ETL job writing to S3 can register partitions itself, so they are queryable in every engine the moment the run ends, with no crawler or extra step:

• Set enableUpdateCatalog to true and pass partitionKeys (matching the table's partition keys, in the same order) — through additional_options on write_dynamic_frame.from_catalog, or as arguments to getSink() followed by setCatalogInfo() and setFormat(). Without enableUpdateCatalog, the job never touches the catalog.
• updateBehavior decides what happens to the schema: UPDATE_IN_DATABASE (the default) adds partitions and overwrites the table schema with the schema the job writes — which can add, drop or retype columns; LOG keeps the catalog schema and adds new partitions only when the written columns match it or are a subset of it.
• It works for S3 targets in JSON, CSV, Avro and Parquet (Parquet through the Glue Parquet writer), and schema updates need a partitioned table.

Because these four methods write real partition entries to the Glue Data Catalog, they serve Athena, Redshift Spectrum and EMR alike — unlike projection, which only Athena understands.

Scenario

A Glue job writes hourly Parquet files to s3://lake/events/dt=YYYY-MM-DD/hr=HH/, and analysts complain new hours don't appear in Athena or Spectrum until the nightly crawler runs. Lowest-latency fixes: turn on enableUpdateCatalog with partitionKeys in the job, or have the writer issue ALTER TABLE ADD PARTITION after each write. If only Athena queries the table, partition projection over the date and hour ranges is also an option — any of these beats scheduling the crawler hourly.

Partition projection and partition indexes both attack slow query planning on heavily partitioned tables, but they work differently: projection makes Athena compute partitions instead of reading them from the catalog, while an index makes the catalog's partition lookup faster.

Partition projection (Athena only)

With 'projection.enabled'='true' and per-column rules in the table properties, Athena calculates partition values and locations at query time. Nothing has to be registered, and planning no longer fetches large partition lists. The rules:

• Types: date (with format, range such as 2022-01-01,NOW, interval), integer (with range, optional interval, and digits for zero padding — digits = 2 produces 01, 02 …), enum (a listed set of values), and injected for values that cannot be listed (such as device IDs). An injected column is never enumerated, so every query must supply its value(s) in the WHERE clause; a query that spans all values without naming them fails.
• Range bounds: values outside the declared range are never generated, so data outside it returns no rows.
• Location: for Hive-style folders Athena builds paths as key=value under the table location. For any other layout, set storage.location.template with a ${column} placeholder per partition key, such as s3://telemetry-logs/${year}/${month}/${day}/; without it Athena looks for Hive-style paths and finds nothing.
• Catalog partitions are ignored: once projection is enabled, Athena does not read the partitions registered in the catalog for that table — so a projection range that starts later than the oldest data hides that older data, even though its partitions are still registered.
• Athena only: Redshift Spectrum, EMR and Glue jobs do not apply projection rules. If they read the same table, partitions must still be registered in the catalog.

Glue partition indexes

A partition index is built on a list of the table's partition keys, so the Data Catalog fetches only matching partitions instead of loading every partition and filtering. They suit tables with very many partitions whose values are not predictable enough for projection, and they work for every engine. The rules:

• An index is used only when the query's filter includes the index's first key (an index on (region, product, dt) does not help a filter on product and dt alone). Supported operators are equality and ranges joined by AND.
• A table can have up to three partition indexes, so add another index whose leading key matches a different query pattern; key names, types and order of indexed keys cannot be changed after creation.
• Crawlers create a partition index for S3 and Delta Lake targets by default — but they do not set partition_filtering.enabled on the table, so Athena will not use that index until you add the property (next rule).
• Athena uses partition indexes only after the table property partition_filtering.enabled is set to true; Redshift Spectrum, EMR and Glue Spark DataFrames use active indexes without extra steps.

A Glue connection is a catalog object that stores how to reach a data store — connection type, endpoint, credentials, and network placement — so crawlers and ETL jobs can use sources and targets that aren't public S3 paths.

JDBC connections cover relational sources and targets: Amazon RDS, Aurora, Amazon Redshift, and on-premises or self-managed databases reachable over JDBC. The connection stores the JDBC URL and credentials — best practice is referencing a secret in AWS Secrets Manager rather than storing a username and password in the connection, so no password lives in a Glue resource and rotation needs no Glue change. Purpose-built connection types also exist for DynamoDB, MongoDB, Kafka/MSK, and a growing set of SaaS sources.

Permissions for the secret. The crawler or job role must read the secret value: secretsmanager:GetSecretValue on the secret, and, when the secret is encrypted with a customer managed KMS key, kms:Decrypt on that key (the AWS managed key needs no key-policy change). secretsmanager:DescribeSecret only returns metadata, not the value, and kms:GenerateDataKey is used when a new secret value is written, not read. Missing read permissions make runs fail with access-denied errors before any network connection is attempted.

Networking is the part candidates get wrong. To reach a database inside a VPC, the connection specifies a VPC, subnet, and security group; when a job or crawler runs, Glue creates elastic network interfaces (ENIs) in that subnet with that security group, and all of its traffic originates from them. Keep three rules apart:

• Self-referencing rule on the connection's security group — a Glue requirement: the group must have an inbound rule allowing all TCP traffic from itself so Glue's components can talk to each other. Glue checks for it when the job or crawler starts and fails up front if it is missing.
• Database security group inbound from the connection's group — the database's security group must allow the database port from the connection's security group (or its subnet). When this is missing, the run starts and then times out opening the connection to the database port — the usual cause of JDBC timeouts.
• Routes from the connection's subnet — because the ENIs live in the connection's private subnet, everything else the job touches must be reachable from there. For Amazon S3 that means an S3 gateway VPC endpoint on the subnet's route table (or a NAT gateway where internet egress is allowed). A bucket policy with aws:SourceVpc or aws:SourceVpce grants permission but provides no route, so it cannot fix a missing endpoint. DNS resolution must also be enabled in the VPC for hostname-based JDBC URLs.

Connections are reused across the system: a crawler uses one to catalog a source database's schema, and ETL jobs use the same connection as a source or a target — for example writing transformed output back to Redshift or Aurora. The console's test-connection action validates reachability and credentials before you debug a failing job blind.

A business data catalog serves people rather than query engines: it makes data assets discoverable, understandable, and governed for the whole organization. On AWS that is Amazon SageMaker Catalog, part of the next generation of Amazon SageMaker (built on Amazon DataZone technology) and surfaced through SageMaker Unified Studio.

In SageMaker Catalog, technical assets — Glue Data Catalog tables, Redshift tables, and other data and ML assets — are enriched with business metadata: plain-language descriptions, business glossary terms, ownership, and quality context. To avoid writing that metadata by hand for hundreds of assets, SageMaker Catalog can generate metadata recommendations with generative AI: draft business names and descriptions for assets and their columns, which the owner reviews, edits and accepts before the asset is published. Work is organized into domains and projects; producers publish assets to the catalog, and consumers search for them and subscribe, triggering an approval workflow. When the owner approves, access is granted automatically through the underlying permission machinery (Lake Formation grants for lake tables, Redshift grants for warehouse tables) — governed self-service instead of tickets and manual bucket policies.

They are complements, not rivals: the business catalog references assets whose schemas live in the technical catalog. Exam cue: schema-for-queries requirements point to the Glue Data Catalog; discovery, business context, and governed subscription workflows point to SageMaker Catalog.