DAGs, scheduling, retries, sensors, best practices
Practise these as flashcards. One or two lines each; the ones you miss come back first.
Start freeA DAG is a Directed Acyclic Graph that represents the workflow structure in Airflow. You define tasks as nodes and set dependencies using >> or << operators, or methods like set_upstream. This ensures tasks run in the correct order.
schedule_interval parameter in a DAG?schedule_interval sets how often the DAG should run, using cron syntax or a timedelta. It determines when Airflow creates new DAG runs and what data interval each run covers.
You can set the retries parameter on the task to specify how many times it should retry, and retry_delay to set the wait time between attempts. This helps handle transient errors automatically.
XComs are stored in Airflow's metadata database and are intended for small pieces of data. Passing large data can slow down the system and bloat the database, so it's better to use external storage for big data.
Catchup is a DAG-level setting that tells Airflow whether to run all missed intervals since the DAG's start date when it's enabled. Backfill refers to the actual process of running those historical DAG runs, either automatically (if catchup is True) or manually triggered.
You can configure email or Slack alerts by setting email, email_on_failure, or on_failure_callback in the DAG or task. For SLAs, you can set the sla parameter and configure sla_miss_callback to trigger alerts when a task exceeds its expected duration.
Sensors are special tasks that wait for an external event, like a file or a database record. Traditional sensors block a worker slot while waiting, but deferrable operators (introduced in Airflow 2.2) use asynchronous triggers, freeing up resources and improving scalability.
I use XComs to pass small pieces of data, like IDs or status flags, between tasks. For larger data, I write it to external storage (like S3 or a database) in one task and read it in the next, passing only references via XCom.
I design tasks so that running them multiple times with the same inputs doesn't cause duplicate or inconsistent results. For example, I use partitioned outputs, upserts, or overwrite operations to make tasks safe to retry.
I use clear and consistent task naming, modularize reusable logic into Python functions or custom operators, avoid deeply nested dependencies, and document the DAG's purpose and schedule. This makes DAGs easier to maintain and debug.
If catchup=True, Airflow will schedule DAG runs for all missed intervals since the last run. If catchup=False, it will only schedule the latest interval, skipping the missed ones. This behavior prevents accidental overload from backfilling unless explicitly desired.
I would check the sensor's configuration, especially the poke_interval and timeout settings, and see if it's using the traditional or deferrable mode. If it's blocking a worker slot, I might switch to a deferrable sensor or adjust the mode to 'reschedule'. I'd also check logs and the external system the sensor is waiting on.