In this article
HDFS: storing data across machines
What is HDFS?
HDFS is the part of Hadoop that stores files by splitting them into blocks and spreading those blocks across many machines. The project describes it as "a distributed filesystem allowing remote callers to read and write data" (Hadoop documentation home, 3.3.5). It runs on ordinary servers and treats hardware failure as normal, not as an exception.
Why it is asked: it opens almost every Hadoop round and sets the vocabulary for the rest. Read the full explanation: data engineering interview preparation.
What do the NameNode and DataNodes do?
A NameNode is the single master server that holds the file-system namespace and decides which DataNode stores which block. DataNodes are the worker servers that store blocks on local disks, serve reads and writes, and create, delete or copy blocks when the NameNode tells them to. User data never flows through the NameNode (HDFS Architecture, 3.3.5).
Why it is asked: it separates metadata from data, the central idea of HDFS.
What is a block, and what is the default block size?
A block is the fixed-size chunk into which HDFS splits every file, and each block is stored and replicated on its own. The default block size is 134,217,728 bytes, which is 128 MiB, and the default replication is 3 (hdfs-default.xml, read October 2026). A 1,000 MiB file becomes eight blocks: seven of 128 MiB and one of 104 MiB.1
Why it is asked: block size drives task counts and NameNode memory.
Why are HDFS blocks so large?
Large blocks keep seek time small compared with transfer time and keep the NameNode's metadata small. Every block is an entry the NameNode must remember, so a file made of many tiny blocks costs memory without adding throughput. Large sequential reads are the workload HDFS targets, which is why the HDFS documentation calls 128 MB (128 MiB) a typical block size.
Why it is asked: it checks that you understand the access pattern HDFS is built for.
How does HDFS replicate blocks across racks?
With a replication factor of three, the default policy places one replica on the writer's node if the writer is on a DataNode (otherwise on a random DataNode in the writer's rack), one on a node in a different rack, and the third on a different node in that same remote rack. Spreading replicas over racks survives a whole-rack failure, and keeping two on one rack limits the cost of cross-rack writes (HDFS Architecture, 3.3.5).
Why it is asked: it tests whether you can reason about failure domains.
How does HDFS notice that a DataNode has died?
Each DataNode sends a heartbeat every three seconds by default (hdfs-default.xml, read October 2026) and a block report listing the blocks it holds. When heartbeats stop, the NameNode marks the node dead after a timeout the documentation gives as more than ten minutes (HDFS Architecture, 3.3.5), then re-replicates its blocks from the surviving copies. With the default settings the timeout works out to 630 seconds.1
Why it is asked: it shows you know how HDFS repairs itself without an operator.
What is safe mode?
Safe mode is the read-only state the NameNode enters at startup, in which it does not replicate or delete blocks. It waits until a configured fraction of blocks, 0.999 by default, has reached its minimum replica count, then leaves after a further 30 seconds and replicates any under-replicated blocks (HDFS Architecture, 3.3.5; hdfs-default.xml).
Why it is asked: it is a common operations question after a restart.
What are the FsImage, the edit log and checkpointing?
The FsImage is a snapshot of the namespace and the edit log is the journal of changes made since. Checkpointing merges the edits into a new FsImage so that a restart does not replay a long log. A checkpoint runs every 3,600 seconds or after 1,000,000 transactions, whichever comes first (hdfs-default.xml, read October 2026).
Why it is asked: it checks that you know what the NameNode persists.
HDFS: availability and cost
How is the NameNode made highly available?
Hadoop runs an active NameNode and one or more standbys that stay in sync through a quorum of JournalNodes. Before Hadoop 2.0.0 the NameNode was a single point of failure, and a crash took the cluster down until an operator restarted it. The active NameNode writes each change to a majority of JournalNodes, so three JournalNodes tolerate one failure and five tolerate two (HDFS High Availability, 3.3.5).1
Why it is asked: single points of failure are a standard senior question.
What is fencing, and what does ZKFC do?
Fencing stops a former active NameNode from acting after a failover, which prevents two nodes from both believing they lead. JournalNodes already accept writes from only one NameNode, and fencing methods such as sshfence add protection against stale reads. The ZooKeeper Failover Controller (ZKFC) runs beside each NameNode, watches its health and competes for a lock to trigger automatic failover (HDFS High Availability, 3.3.5).
Why it is asked: it tests whether you understand split-brain.
What is erasure coding, and why use it instead of replication?
Erasure coding is a way of protecting data by storing computed parity blocks instead of full copies. Three-way replication carries 200% storage overhead, while a Reed-Solomon (RS) scheme with six data and three parity blocks (RS-6-3) carries 50%. A six-block file takes 18 blocks replicated but 9 erasure coded (HDFS Erasure Coding, 3.3.5). The trade is more network and compute on reads and repairs.1
Why it is asked: it separates candidates who know Hadoop 3 from Hadoop 2.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →YARN: sharing the cluster
What is YARN?
YARN is the Hadoop layer that shares cluster resources among applications. The project calls it "a distributed job submission/execution engine allowing remote callers to submit arbitrary work" (Hadoop documentation home, 3.3.5). It lets MapReduce and other frameworks run on the same machines, because scheduling is separate from any one processing model.
Why it is asked: it explains why Hadoop 2 was more than faster MapReduce.
What do the ResourceManager, NodeManager and ApplicationMaster do?
The ResourceManager is the cluster-wide authority that arbitrates resources among all applications. A NodeManager is the per-machine agent that launches and watches containers. An ApplicationMaster is the per-application component that asks the ResourceManager for containers and works with NodeManagers to run and monitor its tasks (YARN documentation, 3.3.5).
Why it is asked: it is the standard YARN architecture question.
What is a container in YARN?
A container is the unit of resources, memory, processor and similar quantities, that the scheduler grants to an application. The YARN documentation describes it as an abstract notion of a resource container that bundles memory, central processing unit (CPU), disk and network. Each task runs inside one, which lets the NodeManager monitor usage (YARN documentation, 3.3.5). Enforcing limits depends on configuration: CPU limits need Linux control groups (cgroups) enabled through the LinuxContainerExecutor (Using CGroups with YARN, 3.3.5).
Why it is asked: it checks the resource-allocation model.
How does the Capacity Scheduler share a cluster?
The Capacity Scheduler divides a cluster into hierarchical queues, each with a reserved percentage of capacity, so several teams can share one cluster. Free resources can flow to any queue beyond its reserved share, bounded by a maximum capacity, and preemption can reclaim containers from queues that exceed their share (Capacity Scheduler, 3.3.5).
Why it is asked: multi-tenant fairness is a daily operations problem.
MapReduce: processing data
How does MapReduce process data?
MapReduce is a programming model that processes data in two user-written steps, map and reduce, with a framework-managed shuffle between them. One map task runs per input split, and map tasks turn input records into intermediate key-value pairs, the framework groups the pairs by key, and reduce runs once per key group (MapReduce Tutorial, 3.3.5).
Why it is asked: it is the foundation of the Hadoop processing story.
Text description of the diagram
- One map task runs per input split.
- Shuffle and sort group the pairs by key.
- The reduce function runs once per key group.
Walk through word count in MapReduce.
Word count maps each line to pairs of (word, 1), groups them by word, and sums the ones per word. For the tutorial's two sample lines, "Hello World Bye World" and "Hello Hadoop Goodbye Hadoop", the mappers emit eight pairs and the output is Bye 1, Goodbye 1, Hadoop 2, Hello 2, World 2 (MapReduce Tutorial, 3.3.5).1
Why it is asked: interviewers want you to trace data, not recite definitions.
What are the shuffle and sort phases?
The shuffle is the step in which each reducer fetches its partition of every mapper's output over the network protocol HTTP (Hypertext Transfer Protocol), and the sort groups that input by key. The two overlap, since fetched outputs are merged while still arriving. A partitioner decides which reducer receives each key, and the default is a hash partitioner (MapReduce Tutorial, 3.3.5).
Why it is asked: shuffle is where much job time and most skew problems live.
What is a combiner?
A combiner is an optional local reducer that runs on each mapper's output before the shuffle, cutting the data sent to reducers. In word count the reducer class doubles as the combiner (MapReduce Tutorial, 3.3.5). It is valid only if the operation is commutative and associative, its output types equal the map output types, and the result is the same whether it runs zero, one or many times. Averaging fails the test unless the combiner carries (sum, count) pairs and the reducer divides.
Why it is asked: it tests whether you know when a combiner is valid.
Hadoop in practice
Is Hadoop still relevant, and how does it relate to Spark?
Hadoop is still maintained: the Apache releases page lists version 3.5.0 from April 2026 and 3.4.3 from February 2026 (Apache Hadoop releases, read October 2026). HDFS and YARN are still maintained and deployed as storage and scheduling layers, while newer engines such as Spark often replace MapReduce for processing. Practices differ by employer, so say which parts you have used.
Why it is asked: it checks that your knowledge is current and honest.
Follow-up questions
How do I prepare for a Hadoop interview?
Be able to draw the HDFS read and write path, the YARN application flow and one MapReduce job on a whiteboard, and quote the default block size and replication factor. Then prepare one failure story, such as a DataNode loss or a skewed reducer. For the wider loop, see how interview loops are structured.
What is the difference between Hadoop and a cloud data warehouse?
Hadoop stores files you manage on clusters you run, while a warehouse hides storage and compute behind Structured Query Language (SQL). Interviewers ask this to hear trade-offs: control and cost at scale against operations effort. Compare with batch, streaming and real-time processing and data modeling questions.
Does Hadoop run on Kubernetes?
Not natively: Apache Hadoop ships no Kubernetes resource manager, and its own scheduler is YARN, which can launch Docker containers through the Linux Container Executor (Launching Applications Using Docker Containers, 3.3.5). Running the Hadoop daemons as Kubernetes pods relies on third-party charts or operators that the Hadoop documentation does not describe, so say that plainly in an interview. The container side is covered in Kubernetes interview questions.
- HDFS splits files into 128 MiB blocks by default, replicated three times, with the NameNode holding metadata and DataNodes holding data.
- Heartbeats and block reports let the NameNode detect dead nodes and re-replicate.
- High availability needs an active and a standby NameNode and at least three JournalNodes, an odd number being recommended.
- Erasure coding cuts overhead from 200% to 50% for cold data at the cost of reconstruction work.
- YARN separates resource management from processing, and MapReduce is map, shuffle and sort, then reduce.
Sources
- HDFS Architecture (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- Apache Hadoop 3.3.5 documentation home — Apache Software Foundation, 15 Mar 2023
- Launching Applications Using Docker Containers (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- Apache Hadoop YARN (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- MapReduce Tutorial (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- HDFS Erasure Coding (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- HDFS High Availability Using the Quorum Journal Manager (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- Using CGroups with YARN (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- Hadoop Capacity Scheduler (Apache Hadoop 3.3.5) — Apache Software Foundation, 15 Mar 2023
- DatanodeManager.java (Apache Hadoop 3.3.5 source) — Apache Software Foundation (date = read date), 25 Oct 2026
- hdfs-default.xml (Apache Hadoop 3.3.5) — Apache Software Foundation (date = read date), 24 Oct 2026
- Apache Hadoop releases — Apache Software Foundation (date = read date), 24 Oct 2026
