All Products
Search

This Product

    Elastic Compute Service:Deploy a Spark cluster on eRDMA-enhanced ECS instances

    Document Center

    Elastic Compute Service:Deploy a Spark cluster on eRDMA-enhanced ECS instances

    Last Updated:Aug 21, 2026

    Buat Spark cluster menggunakan Instance ECS yang ditingkatkan dengan eRDMA dan jalankan Benchmark untuk mengukur kinerja pemrosesan beban dengan latensi ultra-rendah.

    Latar Belakang

    Benchmark adalah alat benchmarking kinerja yang mengukur waktu eksekusi beban, laju transmisi, throughput, dan pemanfaatan resource.

    Langkah 1: Siapkan lingkungan

    Siapkan kluster Hadoop, instal driver yang diperlukan, dan konfigurasikan eRDMA sebelum menjalankan pengujian Benchmark.

    1. Siapkan lingkungan Hadoop. Lewati langkah ini jika kluster Hadoop sudah tersedia.

      • Kebutuhan perangkat keras dan perangkat lunak

        Siapkan versi Hadoop, versi Spark, dan Instance ECS berikut:

        • Versi Hadoop: Hadoop 3.2.1.

        • Versi Spark: Spark 3.2.1.

        • Instance ECS

          • Tipe instans: Lihat Overview.

          • vCPU per instans: 16.

          • Jumlah instans: empat—satu node master dan tiga node pekerja.

      • Instalasi

    2. Login ke node master. Lihat Login ke instans Linux menggunakan Workbench.

    3. Konfigurasikan eRDMA.

      • Instal driver yang diperlukan. Lihat Aktifkan eRDMA pada instans tingkat enterprise.

      • Konfigurasikan pengaturan jaringan.

        1. Buka file hosts:

          vim /etc/hosts

        2. Tekan I untuk masuk ke mode Insert dan ubah isi file:

          192.168.201.83 poc-t5m0        master1
192.168.201.84 poc-t5w0
192.168.201.86 poc-t5w1
192.168.201.85 poc-t5w2
          Catatan

          Ganti alamat IP dengan alamat antarmuka eRDMA (ERI) aktual Anda.

        3. Tekan Esc, masukkan :wq, lalu tekan Enter untuk menyimpan dan keluar.

      • Konfigurasikan pengaturan YARN.

        Catatan

        Lewati sublangkah ini jika NIC default dari Instance ECS mendukung eRDMA.

        1. Buka file yarn-env.sh:

          cd /opt/hadoop-3.2.1/etc/hadoop
vim yarn-env.sh

        2. Tekan I untuk masuk ke mode Insert dan tambahkan konten berikut:

          RDMA_IP=`ip addr show eth1 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1`
export YARN_NODEMANAGER_OPTS="-Dyarn.nodemanager.hostname=$RDMA_IP"
          Catatan

          Ganti eth1 dengan nama ERI aktual Anda.

        3. Tekan Esc, masukkan :wq, lalu tekan Enter untuk menyimpan dan keluar.

      • Konfigurasikan Spark.

        Catatan

        Lewati sublangkah ini jika NIC default dari Instance ECS mendukung eRDMA.

        1. Buka file spark-env.sh:

          cd /opt/spark-3.2.1-bin-hadoop3.2/conf
vim spark-env.sh

        2. Tekan I untuk masuk ke mode Insert dan tambahkan konten berikut:

          export SPARK_LOCAL_IP=`/sbin/ip addr show eth1 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1`
          Catatan

          Ganti eth1 dengan nama ERI aktual Anda.

        3. Tekan Esc, masukkan :wq, lalu tekan Enter untuk menyimpan dan keluar.

    4. Jalankan HDFS dan YARN:

      $HADOOP_HOME/sbin/start-all.sh

    Langkah 2: Unduh paket instalasi Benchmark

    Unduh dan ekstrak paket instalasi Benchmark untuk mendapatkan plug-in Spark eRDMA dan dependensinya.

    1. Unduh paket instalasi Benchmark:

      wget https://mracc-release.oss-cn-beijing.aliyuncs.com/erdma-spark/spark-erdma-jverbs.tar.gz

    2. Ekstrak paket spark-erdma-jverbs.tar.gz:

      tar -zxvf spark-erdma-jverbs.tar.gz

      Paket tersebut berisi komponen-komponen berikut:

      • erdmalib: library native yang dibutuhkan oleh plug-in spark-erdma, sesuai dengan libdisni.so.

      • plugin-sparkrdma: plug-in Spark RDMA dan library dependensi, sesuai dengan spark-eRDMA-1.0-for-spark-3.2.1.jar dan disni-2.1-jar-with-dependencies.jar.

    Langkah 3: Jalankan pengujian Benchmark

    Konfigurasikan entri rute IP dan pengaturan Spark, hasilkan data uji, lalu jalankan pengujian Benchmark.

    1. Ubah entri rute IP.

      Catatan

      Lewati langkah ini jika NIC default dari Instance ECS Anda mendukung eRDMA.

      route del -net 192.168.201.0 netmask 255.255.255.0 metric 0 dev eth0 && \
route add -net 192.168.201.0 netmask 255.255.255.0 metric 1000 dev eth0
      Catatan

      Ganti alamat IP dengan alamat gateway ERI aktual Anda.

    2. Konfigurasikan Spark.

      1. Buka file spark-jverbs-erdma.conf:

        vim /opt/spark-3.2.1-bin-hadoop3.2/conf/spark-jverbs-erdma.conf

      2. Tekan I untuk masuk ke mode Insert dan ubah isi file:

        spark.master yarn
spark.deploy-mode client
#driver
spark.driver.cores 4
spark.driver.memory 19g
#executor
spark.executor.instances 12
spark.executor.memory 10g
spark.executor.cores 4
spark.executor.heartbeatInterval   60s
#shuffle
spark.task.maxFailures 4
spark.default.parallelism 36
spark.sql.shuffle.partitions 192
spark.shuffle.compress            true
spark.shuffle.spill.compress      true

#other
spark.network.timeout 3600
spark.sql.broadcastTimeout 3600
spark.eventLog.enabled             false
spark.eventLog.dir                 hdfs://master1:9000/sparklogs
spark.eventLog.compress            true
spark.yarn.historyServer.address   master1:18080
spark.serializer                  org.apache.spark.serializer.KryoSerializer

#eRDMA
spark.driver.extraLibraryPath   /path/erdmalib
spark.executor.extraLibraryPath   /path/erdmalib
spark.driver.extraClassPath       /path/spark-eRDMA-1.0-for-spark-3.2.1.jar:/path/disni-2.1-jar-with-dependencies.jar
spark.executor.extraClassPath     /path/spark-eRDMA-1.0-for-spark-3.2.1.jar:/path/disni-2.1-jar-with-dependencies.jar
spark.shuffle.manager org.apache.spark.shuffle.sort.RdmaShuffleManager
spark.shuffle.sort.io.plugin.class org.apache.spark.shuffle.rdma.RdmaLocalDiskShuffleDataIO
spark.shuffle.rdma.recvQueueDepth  128
        Catatan

        • Atur spark.shuffle.compress ke false untuk rasio akselerasi yang lebih baik.

        • Kode contoh menggunakan pengaturan resource Spark untuk Instance ECS dengan 32 vCPU dan memori 128 GB. Sesuaikan spark.executor.instances, spark.executor.memory, spark.executor.cores, dan spark.sql.shuffle.partitions berdasarkan skala kluster atau spesifikasi instans Anda.

      3. Tekan Esc, masukkan :wq, lalu tekan Enter untuk menyimpan dan keluar.

    3. Hasilkan data:

      cd /opt/spark-3.2.1-bin-hadoop3.2/conf
spark-submit --properties-file /opt/spark-3.2.1-bin-hadoop3.2/conf/spark-normal.conf --class com.databricks.spark.sql.perf.tpcds.TPCDS_Bench_DataGen spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar hdfs://master1:9000/tmp/tpcds_400 tpcds_400 400 parquet
      Catatan

      400 menentukan volume data dalam GB. Sesuaikan berdasarkan skala kluster Anda.

    4. Jalankan pengujian Benchmark:

      spark-submit --properties-file /opt/spark-3.2.1-bin-hadoop3.2/conf/spark-jverbs-erdma.conf --class com.databricks.spark.sql.perf.tpcds.TPCDS_Bench_RunAllQuery spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar all hdfs://master1:9000/tmp/tpcds_400 tpcds_400 /tmp/tpcds_400_result

      Output berikut menunjukkan bahwa pengujian telah selesai. Hasil pengujian menampilkan waktu eksekusi beban dari Spark cluster.

      22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 371.0 in stage 1.0 (TID 755) in 31 ms on c7re4x004 (executor 1) (372/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 372.0 in stage 1.0 (TID 756) in 29 ms on c7re4x004 (executor 1) (373/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 376.0 in stage 1.0 (TID 760) in 23 ms on c7re4x004 (executor 1) (374/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 374.0 in stage 1.0 (TID 758) in 27 ms on c7re4x004 (executor 1) (375/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 373.0 in stage 1.0 (TID 757) in 31 ms on c7re4x004 (executor 1) (376/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 377.0 in stage 1.0 (TID 761) in 24 ms on c7re4x004 (executor 1) (377/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 375.0 in stage 1.0 (TID 759) in 29 ms on c7re4x004 (executor 1) (378/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 381.0 in stage 1.0 (TID 765) in 53 ms on c7re4x004 (executor 1) (379/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 380.0 in stage 1.0 (TID 764) in 53 ms on c7re4x004 (executor 1) (380/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 379.0 in stage 1.0 (TID 763) in 56 ms on c7re4x004 (executor 1) (381/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 382.0 in stage 1.0 (TID 766) in 55 ms on c7re4x004 (executor 1) (382/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 378.0 in stage 1.0 (TID 762) in 56 ms on c7re4x004 (executor 1) (383/384)
22/06/10 15:35:40 INFO scheduler.TaskSetManager: Finished task 383.0 in stage 1.0 (TID 767) in 53 ms on c7re4x004 (executor 1) (384/384)
22/06/10 15:35:40 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/06/10 15:35:40 INFO scheduler.DAGScheduler: ResultStage 1 (runJob at SparkHadoopWriter.scala:78) finished in 1.599 s
22/06/10 15:35:40 INFO scheduler.DAGScheduler: Job 0 finished: runJob at SparkHadoopWriter.scala:78, took 7.308027 s
22/06/10 15:35:40 INFO io.SparkHadoopWriter: Job job_20220610153533_0002 committed.
22/06/10 15:35:40 INFO server.AbstractConnector: Stopped Spark@1cfd3b14{HTTP/1.1,[http/1.1]}{192.168.145.101:4040}
22/06/10 15:35:40 INFO ui.SparkUI: Stopped Spark web UI at http://c7re4x001:4040
22/06/10 15:35:41 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
22/06/10 15:35:41 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
22/06/10 15:35:41 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
22/06/10 15:35:41 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
22/06/10 15:35:41 INFO cluster.YarnClientSchedulerBackend: Stopped
22/06/10 15:35:41 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/06/10 15:35:41 INFO memory.MemoryStore: MemoryStore cleared
22/06/10 15:35:41 INFO storage.BlockManager: BlockManager stopped
22/06/10 15:35:41 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
22/06/10 15:35:41 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/06/10 15:35:41 INFO spark.SparkContext: Successfully stopped SparkContext
22/06/10 15:35:41 INFO util.ShutdownHookManager: Shutdown hook called
22/06/10 15:35:41 INFO util.ShutdownHookManager: Deleting directory /mnt/data/spark_tmp/spark-d363e35b-badc-43ba-9fa9-476a9301f65c
22/06/10 15:35:41 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-87ef4db1-7c2f-4f23-b989-f27b9bd9ac28
      Catatan

      Untuk membandingkan kinerja eRDMA, hapus konfigurasi plug-in spark-erdma dari direktori konfigurasi Spark atau gunakan Spark cluster tanpa eRDMA, jalankan pengujian Benchmark yang sama, lalu bandingkan hasilnya.