In this article
Architecture and execution
These answers suit candidates preparing for a data engineer interview as well as general Spark roles.
What is Spark's architecture: driver, executors and cluster manager?
The driver is the process that runs the main() function of the application and creates the SparkContext (Apache Spark docs, cluster overview, 3.5.8, accessed October 2026). A cluster manager allocates resources, and executors run the tasks. The SparkContext sends tasks to the executors to run. Each application gets its own executors, which stay up for the whole application and run tasks in multiple threads (cluster overview, latest).
Text description of the diagram
- The driver schedules tasks on the cluster.
- The cluster manager only allocates resources across applications.
- The SparkContext sends tasks to the executors to run.
Why it is asked: every tuning answer depends on knowing which process does what. Read the full explanation: Spark docs, cluster overview.
Who schedules tasks, the driver or the cluster manager?
The driver schedules tasks on the cluster; the cluster manager only allocates resources across applications. The docs add that because the driver schedules tasks, it should run close to the worker nodes, preferably on the same local area network (cluster overview, latest). A driver far from the executors adds delay to every scheduling round trip.
Why it is asked: it corrects the common belief that the cluster manager schedules tasks. Read the full explanation: Spark docs, cluster overview.
What are a job, a stage and a task in Spark?
In Spark's glossary a job is a parallel computation consisting of multiple tasks that gets spawned in response to a Spark action, and a task is a unit of work that will be sent to one executor (cluster overview, 3.5.8). Spark divides each job into stages of tasks that depend on each other; a new stage normally begins where data must be shuffled.
Why it is asked: reading the Spark user interface depends on these three terms. Read the full explanation: Spark docs, cluster overview.
Which cluster managers does Spark support?
The SparkContext can connect to several cluster managers: Spark's own standalone manager, YARN (Yet Another Resource Negotiator) or Kubernetes (cluster overview, latest). The 3.5.8 docs also list Mesos but mark it deprecated, and it was removed in Spark 4.0, so do not present Mesos as a current choice. For the wider ecosystem around YARN, see the Apache Hadoop interview questions.
Why it is asked: it shows whether your Spark knowledge is current. Read the full explanation: Spark docs, cluster overview.
What is the difference between client and cluster deploy mode?
Deploy mode distinguishes where the driver process runs: in cluster mode the framework launches the driver inside the cluster, and in client mode the submitter launches it outside the cluster (cluster overview, 3.5.8). Because the driver schedules tasks, it should run close to the worker nodes (cluster overview, latest), so a driver on a laptop across a slow link favors cluster mode.
Why it is asked: it tests whether you connect where the driver runs to scheduling delay. Read the full explanation: Spark docs, cluster overview.
Can two Spark applications share data in memory?
No. The docs state that data cannot be passed between different Spark applications, meaning separate SparkContext instances, unless it is written to an external storage system (cluster overview, latest). Each application has its own executors, so a cached dataset in one cannot be read by another. Write the shared result to storage instead.
Why it is asked: it tests whether you know the limits of caching. Read the full explanation: Spark docs, cluster overview.
Lazy evaluation and caching
What are transformations and actions in Spark?
A resilient distributed dataset (RDD) is Spark's basic collection of elements partitioned across the cluster. Transformations such as map and filter describe a new dataset; actions such as count or collect return a result to the driver. Lazy evaluation is the rule that transformations do not compute their results right away: they run only when an action requires a result (RDD programming guide, accessed October 2026).
Why it is asked: it is a typical opening Spark question. Read the full explanation: the PySpark tutorial for the Python interface, or the Spark docs, RDD programming guide.
What is the difference between an RDD, a DataFrame and a Dataset?
An RDD is a low-level collection of elements partitioned across the cluster. A DataFrame is a Dataset organized into named columns, so Spark Structured Query Language (SQL) can plan it (Dataset application programming interface (API) docs). Because you describe columns and expressions, the Catalyst optimizer can rewrite the plan and adaptive query execution can re-plan it at runtime (Spark SQL performance tuning); Spark cannot look inside functions passed to RDD operations. The typed Dataset API is for Scala and Java; Python uses DataFrames.
Why it is asked: it tests why the structured API gives the engine more to optimize. Read the full explanation: Spark SQL performance tuning.
What happens if you run three actions on an uncached RDD?
By default each transformed RDD may be recomputed each time you run an action on it (RDD programming guide). So three actions on one uncached dataset recompute its transformations three times, back to the source or to the last shuffle, because Spark "automatically persists some intermediate data in shuffle operations (e.g. reduceByKey), even without users calling persist" (same guide, shuffle operations). In an illustrative lineage with no shuffle, where building the dataset costs 10 units of work, that is 30 units without caching and 10 with it.1
Why it is asked: it checks that you can see hidden repeated work in a pipeline. Read the full explanation: Spark docs, RDD programming guide.
What is the difference between cache and persist?
Both keep a dataset around on the cluster for much faster access the next time you query it (RDD programming guide). The persist method lets you choose a storage level, such as MEMORY_ONLY, MEMORY_AND_DISK or DISK_ONLY; cache uses the default level, and that default differs by API: MEMORY_ONLY for an RDD (2.4.0 guide) but MEMORY_AND_DISK for a Dataset or DataFrame (Dataset API docs). Pick disk-backed levels when the data does not fit in memory.
Why it is asked: it tests whether you choose storage levels deliberately. Read the full explanation: Spark docs, RDD programming guide.
Does cache() mean the data is never recomputed?
No. Cached partitions can be dropped: the RDD guide says Spark drops out old data partitions in a least-recently-used (LRU) fashion (RDD programming guide, removing data). Spark then recomputes a missing partition from its lineage when needed. A memory-only cache saves work but is not a store of record.
Why it is asked: it separates a performance hint from a durability promise. Read the full explanation: Spark docs, RDD programming guide.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Shuffles and joins
What is a shuffle and why is it expensive?
A shuffle is the redistribution of data across partitions so that records with the same key end up together, as in a groupBy or a join. The guide says a shuffle involves disk and network input and output plus data serialization, and creates many intermediate files on disk (RDD programming guide, performance impact). That makes it usually the costliest step of a job. Fewer shuffles, or less data entering them, is the main tuning lever.
Why it is asked: shuffle cost explains most slow-job stories. Read the full explanation: Spark docs, RDD programming guide.
What is the difference between narrow and wide transformations?
A narrow transformation, such as map or filter, computes each output partition from one input partition, so no data moves between executors. A wide transformation, such as reduceByKey or a join, needs data from many input partitions and triggers a shuffle; the RDD guide lists the operations that can cause one (RDD programming guide, shuffle operations). Each shuffle starts a new stage, so counting wide transformations predicts the stages of a job.
Why it is asked: it links lazy plans to stages and shuffles. Read the full explanation: Spark docs, RDD programming guide.
What is the difference between repartition and coalesce?
The repartition method redistributes data into the number of partitions you ask for with a full shuffle, so it can raise or lower the count and evens out partition sizes. The coalesce method merges existing partitions to reduce the count and avoids a full shuffle, so it is cheaper but can leave partitions uneven. The guide lists repartition operations like repartition and coalesce among those that can cause a shuffle (RDD programming guide). Shrink with coalesce before a write; rebalance or grow with repartition.
Why it is asked: it tests whether you weigh shuffle cost against partition balance. Read the full explanation: Spark docs, RDD programming guide.
What is the difference between reduceByKey and groupByKey?
Both gather values by key and both shuffle, but reduceByKey combines values within each partition first, so less data crosses the network. The groupByKey method sends every value across and builds the full list per key, which can exhaust memory on a popular key. The RDD guide says that when you group data in order to aggregate it, reduceByKey or aggregateByKey "will yield much better performance" (RDD programming guide).
Why it is asked: it is a classic test of understanding what a shuffle moves. Read the full explanation: Spark docs, RDD programming guide.
What is a broadcast join, and when does Spark use one?
A broadcast join is a join strategy that sends a copy of the small table to every worker node, so the large table is joined where it sits and is not shuffled. Spark broadcasts a table automatically when its estimated size, taken from statistics rather than file size, is under spark.sql.autoBroadcastJoinThreshold, default 10 megabytes (10,485,760 bytes); -1 disables it (Spark SQL performance tuning). A 5 megabyte table qualifies; a 50 megabyte one needs a BROADCAST hint, or adaptive query execution finding it small at runtime.1
Why it is asked: it tests join strategy and the cost of a shuffle. Read the full explanation: Spark SQL performance tuning.
What is Adaptive Query Execution?
Adaptive Query Execution (AQE) is an optimization that uses runtime statistics to choose the most efficient plan; it has been enabled by default since Apache Spark 3.2.0 (Spark SQL performance tuning, latest). It can convert a sort-merge join to a broadcast hash join when a join side turns out small at runtime, and it can split skewed partitions (performance tuning, latest).
Why it is asked: it shows you know what the engine tunes for you and what it does not. Read the full explanation: Spark SQL performance tuning.
How do you handle data skew in a join?
Skew means a few partitions are far larger than the rest, so one straggler task holds up the stage. AQE detects it from shuffle statistics and splits the skewed partitions into smaller ones (Databricks, Adaptive Query Execution, May 2020). A partition counts as skewed when it is larger than spark.sql.adaptive.skewJoin.skewedPartitionFactor (default 5.0) times the median size and larger than skewedPartitionThresholdInBytes (default 256MB) (Spark SQL performance tuning); with a median of 100 megabytes the bar is 500 megabytes.1 Manual fixes are salting the hot key with a random suffix, broadcasting the small side, or joining the hot keys separately.
Why it is asked: skew is the usual cause of one task running long. Read the full explanation: Spark SQL performance tuning.
Structured Streaming
How does Structured Streaming process data?
By default Structured Streaming queries run on a micro-batch engine, which processes a data stream as a series of small batch jobs, with end-to-end latencies as low as 100 milliseconds (Structured Streaming guide, 3.5.6). It is not record-at-a-time by default. The engine records the offset range processed in each trigger using checkpointing and write-ahead logs (3.5.6 guide).
Why it is asked: it tests whether you know the processing model before you discuss latency. Read the full explanation: streaming vs batch, or the Spark docs, Structured Streaming guide.
How do you get exactly-once results in Structured Streaming?
Use replayable sources and idempotent sinks: together they let Structured Streaming ensure end-to-end exactly-once semantics under any failure (Structured Streaming guide, 3.5.6). Checkpointing records progress so a restarted query replays the same offsets. Exactly-once is a property of the whole pipeline, not of Spark alone: a non-replayable source or a sink that is not idempotent breaks it.
Why it is asked: delivery semantics are a favorite follow-up for pipeline designers. Read the full explanation: Spark docs, Structured Streaming guide.
Is foreachBatch exactly-once, and what about continuous processing?
By default foreachBatch provides only at-least-once writes; you can use the batch id to deduplicate output and reach exactly-once (3.5.6 guide). Continuous processing is an experimental mode, introduced in Spark 2.3, with end-to-end latency as low as 1 millisecond but only at-least-once fault tolerance (3.5.6 guide). Choose it only when that trade is acceptable.
Why it is asked: it tests whether you read delivery semantics precisely instead of assuming the strongest. Read the full explanation: streaming vs batch, or the Spark docs, Structured Streaming guide.
Follow-up questions
What are the common Spark interview questions?
The twenty-one above cover the topics worth knowing: the driver, executors, cluster manager and deploy modes, lazy evaluation, caching, shuffles, joins, skew and streaming delivery semantics. Pair them with Structured Query Language (SQL) practice, since many rounds mix both; the SQL interview questions guide covers that side.
What are Spark data engineer interview questions?
Spark data engineer interview questions usually move from the engine to the pipeline: how a job runs, why a stage is slow, how to fix a skewed join, and how a stream recovers after a failure. Expect design follow-ups on table layout too, so read the data modeling interview questions and the full data engineer interview guide.
What are data engineer interview questions on Spark?
They are the same topics asked in a pipeline setting: choosing a join strategy for a large fact table and a small dimension table, deciding what to cache, and choosing a sink that makes a stream exactly-once. A strong answer names the trade-off, not only the setting.
What are interview questions on Spark configuration in Talend?
The tool-specific questions are covered on their own page, the Talend Spark configuration interview questions; the Spark ideas above still apply.
How should you prepare for a Spark interview?
Run each idea on a small dataset until you can predict what the plan does: build an RDD, call two actions with and without cache, and compare a join before and after a BROADCAST hint. Then say each answer aloud in two or three sentences, with the trade-off last.
- The driver schedules tasks; the cluster manager only allocates resources, and executors run the tasks.
- Transformations are lazy, so an action triggers computation and an uncached dataset may be recomputed for every action.
- A shuffle moves data across the network; broadcast joins and AQE exist to avoid or reduce it.
- Narrow transformations stay within a partition; wide ones shuffle and start a new stage.
- Exactly-once streaming needs a replayable source, checkpointing and an idempotent sink.
- Skew is handled by AQE splitting partitions, or by hand with salting, a broadcast of the small side or a separate join for hot keys.
Sources
- RDD Programming Guide (Apache Spark documentation, latest) — Apache Spark (Apache Software Foundation), accessed 2026-10-26, 26 Oct 2026
- Cluster Mode Overview (Apache Spark documentation, latest) — Apache Spark (Apache Software Foundation), accessed 2026-10-26, 26 Oct 2026
- Cluster Mode Overview (Apache Spark 3.5.8 documentation, glossary) — Apache Spark (Apache Software Foundation), accessed 2026-10-26, 26 Oct 2026
- Performance Tuning, Spark SQL (Apache Spark documentation, latest) — Apache Spark (Apache Software Foundation), accessed 2026-10-26, 26 Oct 2026
- Dataset (Apache Spark Scala API documentation, latest) — Apache Spark (Apache Software Foundation), accessed 2026-10-26, 26 Oct 2026
- Structured Streaming Programming Guide (Apache Spark 3.5.6 documentation) — Apache Spark (Apache Software Foundation), accessed 2026-10-26, 26 Oct 2026
- RDD Programming Guide (Apache Spark 2.4.0 documentation, default storage level) — Apache Spark (Apache Software Foundation), accessed 2026-10-26, 2 Nov 2018
- Adaptive Query Execution: Speeding Up Spark SQL at Runtime — Databricks blog, 29 May 2020
