Orchestration questions in data engineering interviews

In this article

1. Before you start
2. Why interviewers ask this
3. Orchestration from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the data engineer interview questions guide, and the examples use Airflow vocabulary.

Why interviewers ask this

A pipeline that runs once on a laptop is easy; one that must run every day, recover from failures and be rerun for last month without corrupting data is the real job. Orchestration questions separate candidates who have only written scripts from those who have operated them.

A weak answer says "Airflow runs my Python code on a schedule." A strong answer says that the directed acyclic graph (DAG) fixes the order of tasks, that each run belongs to a data interval, and that retries only help if tasks are safe to repeat. A Staff-level answer adds what happens at the edges: late switch-ons, reruns of past dates, and what you may pass between tasks. These are questions an interviewer can ask in a design round such as the senior data engineering system design round.

Orchestration from first principles

Orchestrator

An orchestrator is a scheduling and coordination program that decides when each task in a pipeline runs, in what order, and what happens when one fails, while the tasks themselves do the actual work such as moving or transforming data.

Data interval

A data interval is the span of time whose data one pipeline run is responsible for, such as one calendar day for a daily run; with a data-interval timetable the run is normally created after the interval has ended so that all of that period's data has arrived.

Idempotent task

An idempotent task is a task that has the same effect when run several times with the same input as when run once, so a retry or a rerun cannot create duplicates or change the result.

In plain words

Think of a head chef calling orders in a kitchen. The chef does not chop the vegetables. The chef says who starts when, which dish waits for which, and who redoes a burnt plate. An orchestrator does this for data tasks. The analogy breaks in one place. A kitchen cooks each order once. An orchestrator may be asked to cook the same day's batch again, so every task must be safe to repeat.

In one precise sentence, an orchestrator manages when tasks run, in what order, and how failures are handled, and it does not do the work itself (architecture overview; failure handling is framed the same way in the archived concepts page).

Step 1: A DAG of tasks describes the work, and components run it

A DAG is a directed acyclic graph of tasks that specifies the dependencies between tasks, which defines the order in which they execute (Airflow architecture overview). Tasks describe what to do, such as fetching data or triggering other systems. The kinds of task are operators, sensors and @task functions; a sensor is a special operator that only waits for an external event (archived overview). The DAG is not concerned with what its tasks do; it makes sure they happen at the right time and in the right order (Airflow concepts).

Three services, the workers and one metadata database make this work in Airflow 3. The Dag processor parses Dag files and serializes them into the metadata database, which stores state. The scheduler triggers scheduled workflows and submits tasks to the executor. The executor is a configuration property of the scheduler that runs within the scheduler process, so it is not a separate service; the workers it uses run the tasks, and an application programming interface (API) server serves the user interface (UI) and API (architecture overview). By default, once per minute, the scheduler collects Dag-parsing results and checks whether any active tasks can be triggered (scheduler documentation).

Input: a Python DAG file. Output: a parsed DAG in the metadata database. You now know what is being scheduled; next is when.

Step 2: Catchup decides which missed runs the scheduler creates

Catchup is the DAG setting that decides whether the scheduler creates a run for every data interval missed since the start date or only for the latest one. The Airflow 3 documentation says that by default missed runs are not created (scheduler.catchup_by_default=False) and that the scheduler creates a run only for the latest interval. With catchup=True it starts a run for each interval that has not been run (Dag runs documentation). In Airflow 2 the default is catchup_by_default=True, so a late switch-on there created every missed run unless the DAG set catchup=False (Airflow 2.10 Dag runs).

The interval story below assumes a data-interval timetable, CronDataIntervalTimetable, which is the Airflow 2 default for a cron or preset schedule such as "@daily", and which Airflow 3 uses when create_cron_data_intervals is set to True or the timetable is chosen explicitly (timetable documentation). Under it a run is scheduled after its data interval has ended, so it can collect all the data in that period. The logical date is the start of the data interval, not the time the run executes. A daily run for the first of a month therefore executes on the second. Airflow 3 changed the default timetable; the Trade-offs section shows how that changes the counts.

The worked example below puts numbers on this. Input: a start date and the day you switch the DAG on. Output: the set of runs created.

Step 3: Backfill reruns a chosen range on demand

A backfill is a user-started run of a DAG over an explicit period in the past, used to reprocess historical or missed data. Unlike catchup, which triggers missed runs from the start date through the current data interval, a backfill period is specified explicitly and can include periods before the start date (Astronomer rerun guide). In Airflow 3, backfills are managed by the scheduler and can be triggered through the user interface (UI), the application programming interface (API) or the command-line interface (CLI).

The CLI example in the Airflow 3 backfill documentation names the range and caps concurrent runs with --max-active-runs; Airflow 2 used the command airflow dags backfill instead, which many teams still run:

airflow backfill create --dag-id DAG_ID --from-date START_DATE --to-date END_DATE --reprocess-behavior failed --max-active-runs 3 --run-backwards

The flags choose which runs to redo, how many run at once and in which order. Input: a date range. Output: one run per data interval in that range, rerun with the same logic.

Step 4: Idempotent tasks make retries and reruns safe

A program is idempotent if, for a set input, running it once has the same effect as running it many times. A DAG is idempotent if rerunning the same run with the same inputs has the same effect as running it once (Astronomer best practices). Retries and backfills are safe only when tasks are idempotent.

Two habits give this. Write to the interval's own partition and overwrite it, instead of appending rows. Derive dates from the run's data interval, never from datetime.today(), because that value follows the current date and you cannot rerun a failed run for a past date. You now have the whole picture: a DAG, a scheduler that creates runs, catchup and backfill to fill gaps, and tasks you can safely repeat.

1 · DAG2 · Scheduler3 · Backfill4 · Safe rerunReplay

Step-by-step description

1. A DAG file defines tasks and order.
2. One run is created per data interval.
3. A backfill names its own date range.
4. Tasks overwrite their own partition.

• A DAG file specifies tasks and their order.
• The scheduler creates one run per data interval.
• A backfill request names an explicit date range.
• An idempotent task overwrites its interval partition, so reruns match one run.

A DAG file specifies tasks and their order. The scheduler creates one run per data interval. A backfill request names an explicit date range. An idempotent task overwrites its interval partition, so reruns match one run.

Worked example

Take a daily DAG on a data-interval timetable (CronDataIntervalTimetable, the Airflow 2 default for a cron schedule) with a start date of 2026-01-01, switched on at 08:00 on 2026-01-31 (illustrative dates).¹ The interval for 2026-01-30 ended at midnight, and the interval for 2026-01-31 is still open. With catchup off, the scheduler creates one run, with logical date 2026-01-30.¹ With catchup on, it creates one run for each ended interval, 30 runs with logical dates from 2026-01-01 to 2026-01-30.¹ The regular run with logical date 2026-01-31 then fires on 2026-02-01, after its interval ends.¹

With catchup off the scheduler creates 1 run; with catchup on it creates 30 runs.¹

Text description of the diagram

• With catchup off the scheduler creates 1 run.
• With catchup on it creates 30 runs.

If those 30 runs are a backfill with the documentation example's cap of 3 concurrent runs, they finish in 10 waves if each run takes the same time.¹ Now the duplicate demo, with 1,000 rows per run (illustrative). An append load run three times leaves 3,000 rows, because each retry adds the interval again; an overwrite of that interval's partition leaves 1,000.¹ The first behaves like the problem the definition of idempotent describes, the second like the fix.

Trade-offs and edge cases

Catchup and backfill are different tools. Catchup is automatic and starts from the start date; a backfill is explicit and can reach before the start date.

Text description of the diagram

• Catchup is automatic and starts from the start date.
• A backfill is explicit and can reach before the start date.

Edge cases to name in an answer:

• Passing data between tasks. A cross-communication message (XCom) is a small value one task stores for another. The default backend keeps XComs in the Airflow database, which works for small values but causes trouble with large values or high volume (XCom documentation). Pass a file path or table name, not a dataframe.
• Top-level code. Airflow executes all code in the dags folder on every min_file_process_interval, which defaults to 30 seconds, so a database call at the top of a DAG file runs 2,880 times a day.¹ Code inside a task runs only when the task runs (Astronomer best practices).
• Airflow 2 to 3 change. In Airflow 3.0 the default of create_cron_data_intervals became False, so a cron or preset schedule such as "@daily" is read with CronTriggerTimetable (upgrade guide; timetables). The run fires at the trigger time, the logical date equals the run time and the data interval has zero length. On Airflow 3 defaults the worked example gives 31 runs with catchup on, one per midnight from 2026-01-01 to 2026-01-31, and with catchup off no run is created at switch-on; the first run fires at 2026-02-01 00:00 (timetables).¹ "Logical date is the start of the data interval" holds for data-interval timetables only.
• Retries are not safety. A retry reruns the task; without idempotency it repeats side effects.
• Other scheduling topics. The default trigger rule is all_success, so a task runs only when all its upstream tasks succeeded (concepts). A sensor waits by occupying a worker slot unless it is deferrable, in which case it hands the wait to a lightweight trigger process (deferrable operators). Asset-based (dataset) scheduling starts a Dag when upstream data is updated instead of on a clock, an alternative to time-based runs (asset scheduling).
• Heavy compute. The orchestrator coordinates; the heavy work belongs in an engine such as Spark (see the Spark interview questions) or the warehouse.
• Batch versus stream. Interval-based runs fit batch work; for continuous data read streaming versus batch.

How to answer it in an interview

The spoken structure: define the orchestrator, say what a run is, answer the late switch-on with catchup, then close on idempotency.

Common mistakes

• "The logical date is when the DAG runs." (wrong) With a data-interval timetable the logical date is the start of the data interval, so the run for the first executes on the second; with a trigger timetable, the Airflow 3 default for cron schedules, it equals the run time. "On a data-interval timetable, the run for the first executes on the second." (right)
• "Airflow processes the data." (wrong) The DAG decides order and timing, not what tasks do. "The orchestrator coordinates; Spark or the warehouse does the heavy work." (right)
• "Catchup and backfill are the same." (wrong) One is automatic, the other explicit. "Catchup starts from the start date; a backfill names its own range." (right)
• "Retries make the pipeline safe." (wrong) They repeat side effects. "Retries are safe because my tasks are idempotent." (right)
• "I pass the dataframe through XCom." (wrong) XComs are for small values. "I pass the path of the file I wrote." (right)

Follow-up questions

What is the difference between catchup and backfill in Airflow?

Catchup is the scheduler automatically creating missed runs from the start date through the current interval. A backfill is a run you start for an explicit range, which can include dates before the start date.

What is the logical date in Airflow?

With a data-interval timetable it is the start of the run's data interval, not the time the run executes; with the trigger timetable that Airflow 3 uses by default for cron schedules it equals the run time. Before Airflow 2.2 it was called execution_date (Airflow improvement proposal on richer scheduling intervals; Dag runs documentation).

What does idempotent mean for a pipeline task?

Running the task again with the same input has the same effect as running it once. In practice you overwrite the interval's partition instead of appending, and you never read datetime.today() for the data period.

Can you pass data between tasks with XCom?

Only small metadata. The default backend stores values in the metadata database, so write large data to storage and pass the location.

Why should you avoid heavy code at the top of a DAG file?

The scheduler re-parses DAG files regularly, every 30 seconds by default, so top-level calls to external systems run constantly. Put such calls inside tasks.

What if the DAG is switched on a week late instead?

With catchup on, the scheduler creates one run per ended interval, which is 7 runs for a switch-on on 2026-01-08 in the worked example; with catchup off it still creates 1.¹ The count is the number of ended intervals, so it scales with the delay. For the wider interview picture, see the data engineer interview questions guide.

Sources

1. Architecture Overview (Airflow 3 stable) — Apache Airflow documentation, 26 Oct 2026
2. Architecture Overview (Airflow 2.5.3, archived) — Apache Airflow documentation, 26 Oct 2026
3. Dag Runs (stable) — Apache Airflow documentation, 26 Oct 2026
4. Scheduler (stable) — Apache Airflow documentation, 26 Oct 2026
5. XComs (stable) — Apache Airflow documentation, 26 Oct 2026
6. Concepts (Airflow 1.10.9, archived) — Apache Airflow documentation, 26 Oct 2026
7. Upgrading to Airflow 3 — Apache Airflow documentation, 26 Oct 2026
8. Timetables (stable) — Apache Airflow documentation, 26 Oct 2026
9. Backfill (stable) — Apache Airflow documentation, 26 Oct 2026
10. Dag Runs (Airflow 2.10.5) — Apache Airflow documentation, 26 Oct 2026
11. AIP-39: Richer scheduler_interval — Apache Airflow wiki, 26 Oct 2026
12. Deferrable Operators and Triggers (stable) — Apache Airflow documentation, 26 Oct 2026
13. Asset-Aware Scheduling (stable) — Apache Airflow documentation, 26 Oct 2026
14. Rerun Airflow Dags and tasks — Astronomer documentation, 26 Oct 2026
15. DAG writing best practices in Apache Airflow — Astronomer documentation, 26 Oct 2026