An IoT analytics team stores Parquet files in Amazon S3 under paths partitioned by date, hour and device type, which has produced over 400,000 partitions in an AWS Glue Data Catalog table. Athena queries now spend most of their time in planning, and an hourly crawler means new data cannot be queried for up to an hour. The team wants fast planning and new hours to be queryable as soon as files land, with no partition-registration jobs. What should a solutions architect recommend?
Choose one.
Athena partition projection computes partitions from table properties, bypassing catalog partition lookups and registration.
Two requirements must hold together: fast planning on a huge partition count and zero-delay availability without registration. Partition projection meets both because Athena derives partition values from the configured ranges and path template. Partition indexes fix only planning; event-mode crawls and MSCK REPAIR fix only (some of) freshness and still register partitions.
- Split the requirement into planning speed and freshness without registration.
- Test each option against both; most fix one.
- Recognise that only projection removes catalog partitions from the query path.
Exam tip: Highly partitioned, predictable S3 layouts: partition projection gives fast planning and instant new partitions with no crawler.
SAP-C03 Event-Driven, Serverless and Multi-Tenant Architecture — the lesson that teaches this.