The Hadoop Distributed File System (HDFS) is a highly fault-tolerant distributed file system for Hadoop that supports high-throughput access to massive datasets at terabyte or petabyte scale in a cost-effective manner. HDFS is ideal for distributed read and write operations on large datasets, especially in read-heavy scenarios.
Features
HDFS in E-MapReduce clusters offers the following advantages:
-
High fault tolerance and scalability.
-
Shell command interface for direct interaction.
-
Web-based visualization component for easy management.
-
Linux-like file permission management.
-
Locality awareness. HDFS considers the location of compute nodes when allocating storage space.
-
Rebalancing when data is unevenly distributed across DataNodes.
-
Rolling restarts and upgrades.
Architecture
HDFS uses a classic Master/Slave architecture. An HDFS cluster consists of a single NameNode and multiple DataNodes.
The NameNode manages the file system metadata and interacts with clients. DataNodes manage the data blocks stored on them. When a file is uploaded to HDFS, it is divided into one or more blocks. These blocks are distributed across different DataNodes based on the cluster's data backup policy. The NameNode centrally manages the location information of these blocks.
Terms
|
Name |
Description |
|
NameNode |
Manages the file system namespace, maintains the file system directory tree and metadata, and records the mapping between each written block and its parent file. This information is persisted to the local disk as a namespace image (FSImage) and an edit log (EditsLog). |
|
DataNode |
Stores the actual file data. A DataNode stores or serves blocks based on instructions from the NameNode or a client, and periodically reports its block information to the NameNode. |
|
Client |
Accesses the file system by communicating with the NameNode and DataNodes. The client provides a POSIX-like file system interface. |
|
Blocks |
HDFS splits files into 128 MB blocks for storage. These blocks can be stored on different nodes, allowing HDFS to store single files larger than the capacity of any single disk. By default, each block has three replicas. If an EMR core node uses cloud disks, two replicas are used. Replicas are stored on multiple nodes at the block level, which improves data security and allows distributed jobs to leverage local data for computation, reducing network traffic. |
|
Secondary NameNode |
In a non-high availability cluster, a Secondary NameNode process runs by default. It consumes the EditsLog, periodically merges the FsImage and EditsLog to generate a new FsImage file, and reduces the load on the NameNode. |
|
High availability |
In a high availability (HA) cluster, two NameNodes run by default: one Active and one Standby. The Active NameNode handles requests from DataNodes and clients. The Standby NameNode maintains the same up-to-date metadata and is ready to take over if the Active NameNode fails. Upon failure, the Standby NameNode detects the issue and switches to the Active role to continue serving requests from DataNodes and clients. |