E-MapReduce:ESS (Available only for existing users)

Traditional shuffle mechanisms present several challenges:

• In scenarios with large data volumes, shuffle write operations can cause data to spill to disk, leading to write amplification.

• Shuffle read operations generate a high volume of small network packets, which can cause connection reset errors.

• Shuffle read operations involve numerous small I/O requests and random reads, placing a heavy load on disks and CPUs.

• When the number of mappers (M) and reducers (N) reaches the thousands, the total number of network connections (M × N) can prevent a job from completing.

• The NodeManager and the Spark Shuffle Service run in the same process. When shuffle data volumes are extremely large, the NodeManager can restart, which impacts the stability of YARN scheduling.

ESS provides the following advantages:

• It uses a push-style shuffle mechanism instead of a pull-style one, which reduces memory pressure on mappers.

• It supports I/O aggregation, which reduces the number of shuffle read connections from M × N to N and replaces random reads with sequential reads.

• It supports a two-replica mechanism to reduce the probability of fetch failures.

• It supports a compute-storage separation architecture, allowing you to deploy the Shuffle Service in a separate hardware environment that is decoupled from the compute cluster.

• It eliminates the dependency on local disks when you run Spark on Kubernetes.

The following figure shows the architecture of ESS.