Data Operations and Support practice questions

Amazon EventBridge Scheduler is the purpose-built scheduling service, and one-time schedules with time-zone awareness are exactly the capabilities that distinguish it from EventBridge rules.

EventBridge Scheduler supports at() expressions for one-time invocations, lets you set an IANA time zone such as Europe/London so daylight saving transitions are handled for you, and integrates directly with Lambda as a target. EventBridge scheduled rules cannot do one-time schedules and evaluate cron only in UTC, so DST would silently shift the invocation time. The EC2 and Step Functions options both work mechanically but carry ongoing cost and operational burden for what should be a fire-and-forget schedule.

1. Note the two constraints: a one-time invocation and local time zone plus DST correctness.
2. Recall that EventBridge Scheduler supports at() one-time expressions and IANA time zones, while EventBridge rules support neither.
3. Create the schedule with the Lambda function as the target and let Scheduler handle delivery and retries.

Exam tip: One-time schedules and time-zone-aware schedules are EventBridge Scheduler features; classic EventBridge rules are recurring and UTC-only.

In MWAA, the scheduler continuously parses DAG files from S3; a DAG missing from the UI is a parse-time failure, and MWAA surfaces parse errors through scheduler and DAG processing logs in CloudWatch.

MWAA syncs the dags/ prefix of the environment bucket and the Airflow scheduler attempts to import each Python file. If the file has a syntax error, references a package that is not in requirements.txt, or raises at import time, the DAG never registers and therefore never appears in the UI. Those failures are written to the scheduler log group (and DAG processing logs) in CloudWatch, which is why enabling and reading those logs is the standard first troubleshooting step. Worker logs only exist for tasks that actually ran, CloudTrail only proves the object was uploaded, and web server logs cover UI traffic.

1. Confirm the DAG file landed under the correct dags/ prefix in the environment S3 bucket.
2. Open the CloudWatch scheduler and DAG processing log groups for the MWAA environment and search for the DAG file name.
3. Fix the reported import or syntax error (often a dependency missing from requirements.txt) and wait for the scheduler to re-parse.

Exam tip: A DAG that never shows up in the MWAA UI failed to parse; look in the CloudWatch scheduler and DAG processing logs.

Step Functions Standard workflows persist a full execution history that is the primary troubleshooting tool for failed pipelines.

Every Standard execution logs an ordered event stream: state entered, state exited, task scheduled, task failed, and so on, with the JSON input and output of each state and the error name plus cause for failures. The console visualizes this with the failed state highlighted, and the GetExecutionHistory API returns the same data programmatically. That directly answers all three questions in the stem: which state, what input, what error. CloudWatch metrics are aggregates, CloudTrail covers control-plane API calls, and EventBridge is not a store of state-transition detail.

1. Open the state machine in the Step Functions console and select the failed execution.
2. Use the graph view to spot the state highlighted in red, then open its events to read the input, output, error, and cause fields.
3. Optionally call GetExecutionHistory from the CLI or SDK to pull the same event detail for automation or ticketing.

Exam tip: Debug failed Step Functions executions with the per-execution history, which records every state transition with inputs, outputs, and error causes.

The Amazon Redshift Data API lets applications run SQL over HTTPS with IAM or Secrets Manager authentication, removing the need for drivers and persistent connections.

With the Data API, the Lambda function calls ExecuteStatement on the redshift-data client, receives a statement ID immediately, and later fetches results with GetStatementResult or reacts to the completion event. Because the API is a regional HTTPS endpoint, the function does not need to be attached to the cluster VPC, and there are no JDBC/ODBC drivers or connection pools to manage, which is a major fit for short-lived, event-driven callers like Lambda. The asynchronous model also sidesteps Lambda timeout pressure for long statements.

1. Grant the Lambda execution role redshift-data permissions and access to the cluster credentials via Secrets Manager or temporary IAM credentials.
2. Call ExecuteStatement (or BatchExecuteStatement) with the SQL, cluster identifier, and database.
3. Poll DescribeStatement or subscribe to the Data API completion event, then read results with GetStatementResult if needed.

Exam tip: For serverless or driverless SQL access to Redshift, especially from Lambda, use the Redshift Data API.

Automating AWS services from code means calling the right SDK operation; for Glue ETL jobs the run operation is start_job_run with an Arguments dictionary.

boto3 mirrors the Glue API: StartJobRun starts an existing job definition and returns a JobRunId that can be tracked with get_job_run. Runtime parameters are passed in the Arguments map, and Glue convention prefixes each key with two dashes so the script can read it via getResolvedOptions. create_job is a definition-time call, start_crawler targets crawlers rather than jobs, and lambda_client.invoke addresses Lambda functions only.

1. Create a Glue client with boto3.client('glue') inside the Lambda function.
2. Call start_job_run with JobName and an Arguments map whose keys use the -- prefix.
3. Capture the returned JobRunId and monitor the run with get_job_run or a downstream EventBridge job state-change event.

Exam tip: start_job_run (with an Arguments map of --prefixed parameters) is the boto3 call that executes an existing Glue job.

AWS Glue workflows are the built-in orchestration layer for Glue-native pipelines, wiring crawlers and jobs together with triggers and exposing a single run view.

A Glue workflow starts from a scheduled trigger, runs the crawler, and uses conditional (on-success) triggers to fire each ETL job only after the previous component succeeds. Workflow run properties can even pass state between jobs. Because the entire chain lives in Glue, the console shows one graph with per-node status, satisfying the single-unit monitoring requirement without introducing another orchestration service. MWAA and Step Functions are appropriate when the pipeline spans services beyond Glue; staggered timers are brittle; DataBrew solves a different problem.

1. Create a Glue workflow and add a scheduled trigger as its starting point.
2. Add the crawler, then chain the three jobs with on-success conditional triggers.
3. Monitor workflow runs from the Glue console graph and alert on failed nodes.

Exam tip: For pipelines made only of Glue crawlers and jobs, Glue workflows with triggers provide orchestration without an external orchestrator.