EMR Spark on ACK Product Introduction

Open source big data community&the sixth issue of AliCloud EMR series live broadcast

Subject: EMR spark on ACK product demonstration and best practices

Lecturer: Shi Lei, technical expert of Alibaba Cloud EMR team

Content framework:

• Yunyuan Biochemical Challenge and Alibaba Practice

• Spark containerization scheme

• Product introduction and demonstration

Live playback: scan the QR code at the bottom of the article to join the pin group to watch the playback, or enter the link https://developer.aliyun.com/live/246868

1、 Yunyuan Biochemical Challenge and Alibaba Practice

Development trend of big data technology

Yunyuan Biochemistry is facing challenges

Separation of calculation and storage

How to build an HCFS file system based on object storage

• Fully compatible with existing HDFS

• Performance benchmarking HDFS, cost reduction

Shuffle

How to solve ACK mixed heterogeneous models

• Heterogeneous models have no local disk

• Community [Spark-25299] discussion, support Spark dynamic resources, and become industry consensus

Caching scheme

How to effectively support cross-machine room and cross-private line hybrid cloud

• Need to support the cache system in the container

ACK scheduling

How to solve the scheduling performance bottleneck

• Performance benchmarking Yarn

• Multi-level queue management

other

• Peak shifting dispatching

• Yarnon ACK node resource mutual awareness

Alibaba Practice - EMR on ACK

Introduction to the overall scheme

• Submit to different execution platforms through data development cluster/scheduling platform

• Peak shifting scheduling, adjusted according to the business peak and low peak strategy

• Cloud native data lake architecture, strong capacity expansion and contraction of ACK

• On-cloud and off-cloud hybrid scheduling through dedicated lines

• ACK manages heterogeneous model clusters with good flexibility

2、 Spark containerization scheme

Scheme introduction

RSS Q&A

1. Why do I need Remote Shuffle Service?

• RSS makes Spark jobs unnecessary for the Executor Pod to mount cloud disks. Attaching cloud disks is not conducive to scalability and large-scale production practices.

• The size of the cloud disk cannot be determined in advance. If it is too large, it will waste space. If it is too small, it will fail. RSS is specially designed for storage and computing separation scenarios.

• Executor writes the shuffle data into the RSS system, which is responsible for managing the shuffle data. Executor can recycle it when it is idle. [SPARK-25299]

• It can perfectly support dynamic resources and avoid the long tail task of data skew dragging the Executor resources from being released.

2. How about the performance, cost and scalability of RSS?

• RSS is deeply optimized for shuffle, and is specially designed for storage and computing separation scenarios and K8s elastic scenarios.

• For Shufflefetch stage, random read in reduce stage can be changed into sequential read, which greatly improves the stability and performance of the job.

• You can directly use the disks in the original K8s cluster for deployment without adding extra cloud disks to shuffle. The cost performance is very high, and the deployment method is flexible.

Spark Shuffle

• Generate numMapper * numReducer blocks

• Sequential writing, random reading

• Spill when writing

• Single copy, data loss requires stage recalculation

EMR Remote Shuffle Service

• Additional writing and sequential reading

• Spill without writing

• Two copies; Copy to memory

• The replicas are backed up through the intranet without the need for public bandwidth

RSS TeraSort Benchmark

• Note: Take 10T Terasort as an example, the shuffle volume is about 5.6T after compression. It can be seen that in the RSS scenario, the performance of jobs of this magnitude will be greatly improved because shuffle read is changed to sequential read.