In a Hadoop cluster, data is stored on many machines called DataNodes, which are grouped into Racks. Each rack holds around 30-40 machines.
Hadoop uses a feature called Rack Awareness to improve speed and efficiency. It means the NameNode knows where each DataNode is located (which rack) and uses this to decide where to store data and its copies.
By placing data smartly across racks, Hadoop:
- Reduces network traffic
- Speeds up data access
- Keeps data safe even if a rack fails
This helps the cluster work faster and stay reliable.
Example of Rack in Hadoop cluster:
The image shows a Hadoop cluster with two racks. Each monitor icon inside the racks represents a DataNode, while system outside the racks (top right) represents NameNode. NameNode uses rack information to place data efficiently and ensure fault tolerance.
Understanding Rack-Aware Replica Placement
Hadoop follows specific rules when placing replicas of data blocks across racks. These rules ensure that no single point of failure like a failed DataNode or rack can cause data loss.
This section explains how Hadoop distributes replicas across racks using these rules and why this strategy is important for maintaining a balanced, reliable and high-performing cluster.
In the image, we see 3 racks, each with 4 DataNodes. We are storing 3 file blocks: Block 1 (B1), Block 2 (B2) and Block 3 (B3).
With a replication factor of 3, each block is stored on 3 different DataNodes. Hadoop uses Rack Awareness policies to decide where these replicas go.
Rack Awareness Rules Followed Here:
- No more than 1 replica is placed on the same DataNode
- No more than 2 replicas of a block are on the same rack
- Replicas are distributed across multiple racks for fault tolerance
Example from the image:
- Block 1: Node 1 (Rack 1), Nodes 5 & 6 (Rack 2)
- Block 2: Node 6 (Rack 2), Nodes 10 & 11 (Rack 3)
- Block 3: Nodes 2 & 3 (Rack 1), Node 12 (Rack 3)
This placement ensures high availability, network efficiency and rack-level fault tolerance.
Benefits of Rack Awareness in Hadoop
- Improves Fault Tolerance: Replicas are stored on different racks, so data remains safe even if a rack fails.
- Boosts Network Efficiency: Data transfer within the same rack is faster, reducing network congestion.
- Enhances Performance: Smart replica placement leads to quicker data access and better cluster performance.
- Supports High Availability: Even if some DataNodes go down, data remains available from other replicas.
Real-World Example of Rack-Aware Block Distribution in Hadoop
This example shows how Hadoop handles a real file, Test-File.txt, by splitting it into blocks (A, B, C) and distributing those blocks across DataNodes in different racks.
You can also see important components like NameNode, Standby NameNode and Resource Manager, giving a complete view of how Rack Awareness works in a live Hadoop environment to ensure data availability and fault tolerance.
Rack 1:
- Contains the active NameNode
- Stores Block A on DataNode 1
- Stores Blocks C and A on DataNode 2 (replicas)
Rack 2:
- Has the Standby NameNode
- Stores Block A on DataNode 10
- Stores Block B on DataNode 11 and Y
Rack 3:
- Includes the Resource Manager
- Stores Block B on DataNode 20
- Stores Block C on DataNode 21 and Z
It shows:
- Each block is stored on multiple racks (replication factor = 3)
- Rack Awareness ensures replicas are not placed on same rack unnecessarily
- Roles like NameNode, Standby NameNode and Resource Manager are clearly assigned to different racks, improving fault tolerance and load distribution
