Data engineer interview questions: the complete guide

In this article

1. The vocabulary of data engineering
2. What data engineering rounds test
3. Start here: the learning path
4. Data modeling: facts, dimensions and grain
5. Batch and streaming
6. Spark: partitions, shuffles and joins
7. Quality, idempotent loads and backfills
8. Orchestration
9. Warehouse, lakehouse and lake
10. SQL for data engineers
11. Senior data engineering system design
12. Questions interviewers ask on data engineering
13. All 11 guides in Data engineering
14. Follow-up questions
15. Sources

The vocabulary of data engineering

Data engineering

Data engineering is the work of building and running the systems that collect data from many sources, clean and reshape it, and deliver it to the places where analysts, products and models use it, so that the data is correct, fresh enough and cheap to keep.

Idempotent load

An idempotent load is a load that leaves the destination in the same state whether it runs once or several times, so a retry or a re-run never duplicates or corrupts data. It is the property that makes orchestrator retries and backfills safe to use.

Grain

Grain is the level of detail that one row of a fact table represents, such as one order line or one page view. The Kimball Group says it must be declared before choosing dimensions or facts, because every candidate dimension and fact has to be consistent with it.

Backfill

A backfill is a re-run of a pipeline over past time periods, either to fix an error or to fill in history after a change in logic. Backfills are safe when each run replaces, rather than adds to, what is already there.

What data engineering rounds test

Data engineering rounds test a small set of ideas that every pipeline depends on. We group them into eight topics, one section each: how to shape data for analysis, how to handle data that never stops arriving, how a distributed engine moves data, how to make re-runs safe, how to schedule and recover work, where the data lives, how to write the SQL, and how to design the system that ties it together.

Text description of the diagram

• Data engineering rounds test a small set of ideas that every pipeline depends on.
• The eight topics are data modeling, batch and streaming, Spark, quality and backfills, orchestration, where the data lives, SQL, and senior system design.

Start here: the learning path

Begin with how data is shaped (modeling and grain), then how it arrives (batch, streaming, event time) and how an engine moves it (Spark partitions and shuffles). Next come the ideas that keep it correct (idempotent loads, backfills, orchestration), where it is stored, the SQL that reads it, and finally the system design round that combines all of them.

1. Data modeling interview questions on facts, dimensions, grain and star schemas
2. How do you answer a senior DE system design round?
3. Streaming vs batch: what is the difference and when do you use each?
4. Spark interview questions on caching, shuffles, joins, skew and streaming
5. Data quality, idempotency, backfills: safe pipeline reruns
6. Data warehouse vs lakehouse vs lake: what is the difference?
7. Orchestration questions in data engineering interviews
8. What SQL do data engineers need to know for interviews?

Data modeling: facts, dimensions and grain

Data modeling is the art of laying out tables so that analysis is simple and fast. In the dimensional approach described by the Kimball Group, a fact table holds measurements (in a transaction fact table, each row is one measurement event at the declared grain), and dimension tables hold the descriptions, such as customer or product, that you slice by; a star schema joins one fact table to its dimensions (Kimball Group on star schemas, read 2026-10-24; on transaction fact tables). A good first question in a modeling round is the grain: "one row per what?" (Kimball Group on grain).

The follow-up is history. Kimball's techniques include slowly changing dimensions: Type 1 overwrites the old value, which Kimball says "destroys history", and Type 2 adds a new row to the dimension with the updated values, which keeps it. A strong answer picks one per attribute and says what question the business needs to ask about the past.

Batch and streaming

Batch processing handles a bounded dataset on a schedule; streaming handles data that keeps arriving. The hard part of streaming is that data arrives late and out of order. Akidau et al. (2015) argue that systems should assume completeness is never known, that new data keeps arriving, and that old data may be retracted, and their Dataflow Model lets practitioners choose their own trade-off among correctness, latency and cost.

The interview vocabulary follows from that: event time is when something happened, processing time is when your system saw it, and an event-time window groups events by when they happened, while a processing-time window groups them by when they arrived, which is simpler but shifts results when data is late. The deciding question is how fresh the result must be and how much lateness you accept.

Spark: partitions, shuffles and joins

In Spark, a partition is one chunk of a dataset that a single task processes; a job runs fast when the work is spread evenly over many partitions. A shuffle is Spark's way of redistributing data so it is grouped differently across partitions, and it is a common first check when a job is slow. The Spark programming guide (version 4.2.0, read 2026-10-24) says a shuffle "involves disk I/O, data serialization, and network I/O", and lists the operations that can cause one: repartitioning, the by-key operations such as groupByKey and reduceByKey, and joins (Apache Spark).

Not every join shuffles: a table small enough can be broadcast to all worker nodes instead (the default size limit is 10 MB), and the Spark SQL tuning guide (version 4.2.0, read 2026-10-24) says data skew "can severely downgrade the performance of join queries" and that adaptive execution can split skewed tasks (Spark SQL performance tuning). A solid answer to "why is my job slow" is to count the shuffles, reduce them, reduce the data before the shuffle, and check for skew.

Quality, idempotent loads and backfills

An idempotent load matters because orchestrators retry and people re-run. Apache Airflow's best-practices page (version 3.3.2) says tasks "should produce the same outcome on every re-run", warns that a plain insert statement on a re-run can create duplicate rows and recommends an upsert (a write that updates the row if its key already exists and inserts it if not), and advises never reading "the latest available data" in a task, but a specific partition instead (Airflow docs).

The difference is easy to show. A table partition is a different thing from a Spark partition: it is one slice of a table stored separately, usually one per date, which can be dropped or replaced on its own. Take a day with three source rows, 10, 20 and 30, so the correct result is 3 rows and a sum of 60.

A job that appends and is retried once leaves 6 rows and a sum of 120. A job that deletes that day's partition and then inserts leaves 3 rows and a sum of 60 however many times it runs.¹ Delete-then-insert is safe only across completed runs: if the job dies between the delete and the insert, the day reads empty until the retry, so run both in one transaction or swap the partition in atomically. The second job is also what makes a backfill safe: you re-run any past day and the result is replaced, not doubled.

Text description of the diagram

• A job that appends and is retried once leaves 6 rows and a sum of 120.
• A job that deletes that day's partition and then inserts leaves 3 rows and a sum of 60 however many times it runs.

Orchestration

Orchestration is the layer that schedules tasks, orders them by dependency, retries failures and records what ran. The same Airflow page advises against running database access, heavy computation or network calls at the top level of a pipeline definition, because the scheduler reads that code repeatedly; put the work inside tasks. The interview version is "what happens when step three of five fails at 3 a.m.?".

A complete answer has four parts: the task retries a set number of times with a delay, the failed task is idempotent so its retry cannot double what the first attempt wrote, the later steps wait because they depend on the failed one, and an alert goes to a person if the retries run out. This is why idempotent loads and orchestration are taught together.

Warehouse, lakehouse and lake

A data warehouse is a central store of cleaned, structured data organized for reporting and analysis; a data lake is low-cost storage that holds raw data of any kind in open file formats; a lakehouse tries to give the lake the reliability and speed of a warehouse. Armbrust et al., in a paper for the Conference on Innovative Data Systems Research (CIDR) 2021, describe the history: warehouses wrote data with a fixed schema on the way in but coupled compute to storage and could not hold video, audio or text; lakes held any data cheaply with the schema applied on read, but "punted the problem of data quality and governance downstream" (Armbrust et al., read 2026-10-24).

The paper argues for the lakehouse, built on open formats such as Apache Parquet, with support for machine learning and strong performance; note that its authors work at Databricks, a vendor of such systems. In an interview, state the trade-off by what the data is: structured reporting data suits a warehouse, raw and varied data suits a lake, and one platform for both is the lakehouse argument.

SQL for data engineers

SQL rounds test joins, aggregation and window functions under time pressure. A window function is a function that computes a value for each row from a set of related rows without collapsing them into one, such as a running total per customer or the rank of each order within its day. The usual traps are joins that multiply rows because the key is not unique, and aggregates that count the wrong grain, which is why the modeling question comes first. For technique that carries over from other rounds, see coding interviews.

Senior data engineering system design

Design rounds ask you to assemble everything above for a named scenario, such as moving orders from an operational database to a reporting layer. Walk it in order: the sources and how often they change, the grain of the tables, batch or streaming and why, where the data lives, how loads stay idempotent, how the schedule recovers from failure, and how you would notice bad data. Practice the interview shape in system design and machine learning (ML) system design, and for pipelines that feed models read machine learning operations (MLOps).

Questions interviewers ask on data engineering

Follow-up questions

How can I prepare for a data engineering interview?

Learn the eight topics in this order, run a small idempotent pipeline yourself, and prepare one project story with numbers (volume, latency, error rate). Practice SQL under time limits, and prepare your own questions for the interviewer.

What is asked in a data engineer interview?

Expect SQL, a data modeling question, a pipeline design question, questions on a processing engine such as Spark, and a behavioral round. The interview loops page shows how these rounds fit together, and the behavioral hub covers the story round.

What questions should I ask the interviewer in a data engineering interview?

Ask about data volume and freshness needs, how failures and backfills are handled, who owns data quality, and how schema changes are communicated. The questions to ask page gives a general method.

What is idempotency in a data pipeline?

A job is idempotent when running it again produces the same destination state, so retries and backfills are safe. The usual ways to get it are replacing a partition and using upserts with a stable key.

Should I choose batch or streaming?

Choose by the freshness the business needs, not by fashion. If a result that is hours old is acceptable, batch is simpler to build and fix; choose streaming when minutes matter, and accept the cost of handling late data.

Sources

1. Star Schemas and OLAP Cubes — Kimball Group, Dimensional Modeling Techniques (date = read date), 24 Oct 2026
2. Grain — Kimball Group, Dimensional Modeling Techniques (date = read date), 24 Oct 2026
3. Transaction Fact Tables — Kimball Group, Dimensional Modeling Techniques (date = read date), 24 Oct 2026
4. Type 1: Overwrite — Kimball Group, Dimensional Modeling Techniques (date = read date), 24 Oct 2026
5. Type 2: Add New Row — Kimball Group, Dimensional Modeling Techniques (date = read date), 24 Oct 2026
6. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing — Google Research, PVLDB vol. 8 (Akidau et al.), 31 Aug 2015
7. Apache Spark RDD Programming Guide, shuffle operations — Apache Spark documentation, version 4.2.0 (date = read date), 24 Oct 2026
8. Apache Spark SQL performance tuning (broadcast joins, adaptive execution, skew joins) — Apache Spark documentation, version 4.2.0 (date = read date), 24 Oct 2026
9. Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics — CIDR 2021, Armbrust, Ghodsi, Xin and Zaharia (read 2026-10-24), 11 Jan 2021
10. Apache Airflow best practices — Apache Airflow documentation, version 3.3.2 (date = read date), 24 Oct 2026