Hadoop Distributed File System (HDFS) is a distributed file system for Hadoop. It is highly fault-tolerant and supports high-throughput data access. HDFS can process massive amounts of data, at the terabyte or petabyte scale, while minimizing costs. It is suitable for distributed reads and writes of large-scale data, especially in read-intensive scenarios.
Features
HDFS in an E-MapReduce (EMR) cluster provides the following benefits:
High fault tolerance and scalability.
A shell command interface.
A web UI for visual component management.
Linux-like file permission management.
Locality awareness. HDFS considers the locations of compute nodes when it allocates storage space.
Rebalancing when data is unevenly distributed across DataNodes.
Rolling restarts and upgrades.
Architecture
HDFS uses a classic master-slave architecture. Each HDFS cluster consists of one NameNode and multiple DataNodes.
The NameNode manages the metadata of all files and interacts with clients. DataNodes manage the files stored on their nodes. Each file uploaded to HDFS is split into one or more data blocks. Based on the data replication policy of the HDFS cluster, the blocks are distributed across DataNodes, and the NameNode centrally manages their location information.
Basic concepts
Name | Description |
NameNode | The NameNode manages the file system namespace, maintains the file system directory tree and metadata, and records the mapping between each data block and the file to which the block belongs. This information is persisted to local disks in two forms: the file system image (FsImage) and the edit log (EditsLog). |
DataNode | A DataNode is the core HDFS component that stores data blocks. DataNodes store or provide data blocks based on instructions from the NameNode or clients, and periodically report information about the blocks they store to the NameNode. |
Client | A client accesses the file system and communicates with the NameNode and DataNodes. The client provides a POSIX-like file system interface. |
Blocks | HDFS splits files into 128 MB data blocks that may be stored on different nodes. This allows HDFS to store a single file that is larger than the capacity of any individual disk. By default, each block has three replicas. If the EMR core nodes use cloud disks, each block has two replicas. HDFS stores the replicas on multiple nodes. This approach improves data security and allows distributed jobs to use local data for computing, which reduces network transfers. |
Secondary NameNode | For a non-high-availability cluster, a Secondary NameNode process starts by default. The Secondary NameNode consumes EditsLog, periodically merges FsImage and EditsLog, and generates a new FsImage file to reduce the load on the NameNode. |
High availability | For a high-availability cluster, two NameNodes start by default: an active NameNode and a standby NameNode. They assume different roles. The active NameNode handles requests from DataNodes and clients. The standby NameNode maintains the same up-to-date metadata as the active NameNode and is ready to take over if the active NameNode becomes unavailable. If the active NameNode fails, the standby NameNode detects the failure, switches to the active role, and handles requests from DataNodes and clients. |