MLOps

How will you process 100,000 files across multiple servers on Hadoop?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

You will process 100,000 files across multiple servers on Hadoop by packing the small files into a few hundred large input splits, so each map task has real work; the scheduler runs the tasks beside the data¹. The next question is what changes at 100 million files, and the answer is arithmetic about one machine, the NameNode, not about the servers.

In this article

Before you start
Why interviewers ask this
Processing 100,000 files on Hadoop from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This question belongs to the machine learning operations (MLOps) interview questions path, and the storage and pipelines around it are covered in designing the data platform under machine learning. If the machine-learning side is new, start with machine learning algorithms.

Why interviewers ask this

A candidate who answers "spread the files over more servers" misses the cost that decides this question. The question tests whether you see that many small files cost more in tasks and in metadata than the same bytes held in a few large files. Candidates report it in data-engineering and system-design rounds; no public source ties it to one company.

A weak answer lists tools. A strong answer asks the file size first, counts the tasks that one file per task would create, and names the NameNode as the place where file count turns into memory. A Staff-level answer adds the follow-up an interviewer asks next, what happens at 100 million files, and fixes the problem where the files are written rather than where they are read.

Processing 100,000 files on Hadoop from first principles

NameNode: The NameNode is the Hadoop Distributed File System (HDFS) master server that keeps the metadata of every file and block in memory and tells clients which DataNodes hold each block. Its memory use grows with the number of files and blocks, not with the number of bytes stored.
Input split: An input split is the slice of input data that one map task reads, and by default the framework creates one map task per input split. With the default FileInputFormat, a file smaller than a block is a split of its own, so many small files mean many splits and many tasks.
Small-files problem: The small-files problem is the loss of speed and memory that comes from storing and processing very many files far smaller than the HDFS block size. Each file adds a metadata object on the NameNode and a map task to the job, while holding very little real work.

In plain words

Imagine a mail room that must deliver many postcards, an illustrative 100,000 of them. Every courier pays a fixed cost to take a job: sign in, load a bag, walk out. If each courier carries one postcard, the sign-in takes longer than the walk. If each carries a full bag, the sign-in is a small part of the trip.

A map task is the courier and a small file is the postcard. The analogy breaks in one place: couriers do not also write every postcard into a central ledger, but the NameNode does register every file.

In one precise sentence: processing many files on Hadoop is cheap only when the files are grouped into units near the block size.

The reason is that task startup and NameNode memory are paid per file, not per byte.

Packing small files cuts the work from 100,000 tasks to 782, and the tasks still run where the data sits.CC BY 4.0 — source: How will you process 100,000 files across multiple servers on Hadoop?

Step-by-step description

100,000 files make 100,000 map tasks.
The NameNode holds 200,000 objects.
Combined splits leave 782 map tasks.
Tasks run where the blocks are.

One map task per file means 100,000 map tasks for 100,000 small files.
The NameNode holds about 200,000 objects in memory for the 100,000 files.
Combining files into 128 MB splits leaves about 782 map tasks.
Tasks run on the servers that hold the data blocks.

Step 1: One map task per file means 100,000 tasks

One map task per file means 100,000 map tasks for 100,000 small files (illustrative count).¹ The MapReduce tutorial says the framework creates one map task per input split. FileInputFormat, the default, never puts two files in one split, so each small file becomes its own split (Tom White, Cloudera).

Take 100,000 files of 1 MB each, which is 100 GB in all (illustrative sizes).¹ Assume 10 servers with 8 containers (task slots) each, 2 seconds to start a task and a read speed of 50 MB per second (all illustrative). Reading one 1 MB file then takes 20 milliseconds, so the 2 seconds of startup is 100 times the real work and about 99% of each task's time.¹ The 80 slots run the 100,000 tasks in 1,250 waves, an estimated 2,525 seconds, about 42 minutes.¹

Input: 100,000 files (illustrative). Output: 100,000 tasks. You now know the first cost is tasks; the second is memory.

Step 2: The NameNode counts files, not bytes

The NameNode holds about 200,000 objects in memory for the 100,000 files, about 30 megabytes¹. The HDFS design guide and Tom White's small files post describe the cause: the NameNode keeps file and block metadata in memory, and a common rule of thumb is about 150 bytes per file, directory or block object.

Each 1 MB file is one file object and one block object, so 100,000 files make 200,000 objects, or 30,000,000 bytes (illustrative).¹ That is small: this many files do not break a NameNode. The same arithmetic at 100 million files gives 30 GB, and that is where the real problem starts.¹

A small file does not waste a whole block on disk, because a block uses only the bytes it holds. The waste is metadata and tasks. Adding servers adds DataNodes, not NameNode memory, so it does not fix this.

Input: the file count. Output: bytes of metadata.

Step 3: Combine small files into splits near the block size

Combining files into 128 MB splits leaves about 782 map tasks (illustrative inputs)¹. CombineFileInputFormat packs many files into each split (Javadoc), so the 100 GB make 782 splits instead of 100,000.¹ One map task per file gives 100,000 map tasks; the combined job gives 782, a cut of about 99%.¹

The class is abstract, so in practice you use CombineTextInputFormat and set the maximum split size, mapreduce.input.fileinputformat.split.maxsize, to 128 MB. Without a maximum, blocks from the same rack are combined into one huge split (Javadoc). Locality is only partial.

This changes how the job reads, not how the files are stored, so the 200,000 NameNode objects remain. To fix both problems, pack the files themselves. A SequenceFile (file name as key, contents as value) cuts both the NameNode objects and the map tasks, because tasks split the one large file. A Hadoop Archive (HAR), read as a har:// file system (Hadoop Archives guide), cuts NameNode memory only; a job reading a HAR still gets one input per archived file (Tom White, Cloudera). Upstream compaction into block-sized files fixes both.

In an illustrative model, packing 100 GB into block-sized units cuts tasks from 100,000 to 782 and an estimated 2,525 seconds to 45.6 seconds.¹
Quantity	One file per task	Packed into 128 MB units
Data size	100 GB	100 GB
HDFS files plus blocks	200,000	1,564
NameNode memory at 150 bytes per object	30,000,000 bytes	234,600 bytes
Map tasks	100,000	782
Waves on 80 slots	1,250	10
Estimated wall time	2,525 seconds	45.6 seconds

Combining files into block-sized splits removes about 99% of the map tasks.CC BY 4.0 — source: How will you process 100,000 files across multiple servers on Hadoop?

Text description of the diagram

One map task per file gives 100,000 map tasks.
Combining files into 128 MB splits leaves about 782 map tasks.

Step 4: Tasks run on the servers that hold the data

Tasks run on the servers that hold the data blocks, because the scheduler moves computation to the data. The HDFS design guide states the principle: moving computation is cheaper than moving data.

That is why you do not copy all the files to one server and fan out from there. The files already sit on DataNodes across the cluster, and each task reads mostly blocks stored on its own node or rack, since combined splits give partial locality. With an illustrative 782 tasks on 80 slots the job runs in 10 waves, about 45.6 seconds in the same illustrative model, against 2,525 seconds before.¹

The number of reducers is a separate setting, chosen from the size of the output, not from the file count. Compare this batch design with the other modes in batch vs streaming vs real-time inference; general cluster scheduling is covered in Kubernetes for machine learning infrastructure.

Input: 782 splits (illustrative). Output: the job's results.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.