Content

Target customers: head companies in the digital marketing, online education, and Internet entertainment fields. Features released: EMR now supports ESS. It is an extension to optimize the shuffle operations of computing engines. ESS uses push-style shuffle, instead of pull-style shuffle, to reduce the memory pressure caused by mappers. ESS supports I/O aggregation, reduces the number of connections in a shuffle read task from M × N to N, and converts random read to sequential read. ESS uses a two-replica mechanism to reduce the probability of fetch failures. ESS allows the shuffle service to be deployed in a separated hardware environment to support compute-storage separation. ESS eliminates the dependency on local disks when Spark on Kubernetes is used.