📁 What is HDFS Architecture?
HDFS (Hadoop Distributed File System) is the storage layer of the Hadoop ecosystem. It is designed to store very large files reliably across a cluster of machines and provide high throughput access to application data.
🏗️ High-Level Architecture of HDFS
HDFS follows a Master-Slave architecture, and consists of three main components:
| Component | Role |
|---|---|
| NameNode | 🧠 The master that manages the file system namespace and metadata |
| DataNode | 💾 The workers/slaves that store actual data blocks |
| Secondary NameNode | 🛠️ Performs periodic checkpointing of NameNode metadata (⚠️ not a backup or standby NameNode) |
🔍 Key Concepts
1. NameNode (Master)
- Stores metadata only: filenames, directories, block locations, permissions.
- Does not store actual data.
- Keeps everything in memory, so it’s fast but a single point of failure (unless using HA setup).
2. DataNode (Slave)
- Stores actual blocks of data on local disks.
- Periodically sends heartbeats and block reports to the NameNode.
- Responsible for serving read/write requests from clients.
3. Secondary NameNode
- Periodically merges the edit logs and fsimage (metadata) from NameNode.
- Helps reduce recovery time if the NameNode restarts.
- ⚠️ It’s not a hot backup or automatic failover.
📦 HDFS File Storage Strategy
- Files are split into blocks (default 128 MB).
- Each block is replicated (default 3 copies) for fault tolerance.
-
Replicas are placed:
- One on local rack
- One on a different rack
- One on a random DataNode
🧭 Workflow: How HDFS Works
📝 Writing a File
- Client contacts NameNode for metadata (e.g., block locations).
- NameNode provides list of DataNodes for each block.
- Client writes data directly to DataNodes (not through NameNode).
- Data is replicated as per the replication factor.
📖 Reading a File
- Client contacts NameNode for block locations.
- Client reads directly from DataNodes that have the required blocks.
✅ Key Features of HDFS
| Feature | Description |
|---|---|
| Fault Tolerant | Uses replication to handle node failures |
| High Throughput | Optimized for large datasets and streaming reads |
| Scalable | Easily add more nodes for storage and throughput |
| Write Once, Read Many | Optimized for batch processing, not random writes |
| Rack Awareness | Smart placement of replicas for fault isolation |
📊 Summary
“HDFS is a distributed file system that stores data across a cluster in a fault-tolerant and scalable way. It uses a master-slave architecture with a NameNode for metadata and DataNodes for actual storage. Files are split into blocks and replicated across nodes to ensure availability and durability. HDFS is optimized for high throughput, large-scale data processing, and forms the storage backbone of the Hadoop ecosystem.”