To meet the storage requirements in big data scenarios, Alibaba Cloud has launched d1 series instances with local disks.
d1 series instances use local disks, instead of cloud disks, for data storage. This avoids the high costs caused by multiple copies of data generated when cloud disks are used. d1 series instances do not require all data to be transmitted over the network. This improves disk throughput and leverages the advantages of Hadoop in edge computing. Local disks have better storage performance and lower storage unit price than cloud disks. The cost is almost the same as physical hosts.
However, local disks cannot ensure data reliability. Alibaba Cloud provides a multi-replicas data storage policy for cloud disks to ensure data reliability, and you do not have to worry about damaged disks. For local disks, data reliability is ensured by upper-layer software. Disk and node faults require manual troubleshooting.
EMR provides a complete set of automated O&M solutions to help you easily and reliably use instances with local disks, such as d1 series. You do not need to worry about the O&M process, because high data reliability and high service availability are ensured.
The highlights of automated O&M solutions are as follows:
- Highly reliable distribution of required nodes
- Fault monitoring of local disks and nodes
- Automatic assessment of data migration opportunities
- Automatic data migration of faulty nodes and data balancing
- Automatic HDFS data detection
- Network topology optimization