ACK kümelerinde Hadoop OSS SDK, Hadoop S3 SDK veya JindoSDK kullanarak yerleşik PageRank işiyle OSS verilerini okuyan ve yazan Spark işlerini çalıştırın.
Önkoşullar
Aşağıdakilere sahip olduğunuzdan emin olun:
Kubernetes 1.24 veya üzeri bir sürümde çalışan bir ACK Pro kümesi veya ACK Serverless Pro kümesi: ACK yönetilen kümesi oluşturma, ACK Serverless kümesi oluşturma ve ACK kümelerini manuel olarak yükseltme
Yüklü ack-spark-operator eklentisi: ack-spark-operator bileşenini yükleme
Kümeyle bağlantılı bir kubectl istemcisi: kubeconfig dosyasını edinme ve kubectl'i kümeye bağlama
Bir OSS paketi: Paket oluşturma
Yüklenmiş ve yapılandırılmış ossutil: ossutil komut referansı
Bir OSS entegrasyonu seçin
OSS erişimi için üç SDK mevcuttur. Devam etmeden önce birini seçin.
| SDK | Ne zaman kullanılır | URI şeması |
|---|---|---|
| Hadoop OSS SDK | Aliyun dosya sistemi aracılığıyla yerel OSS desteği; çoğu iş yükü için basittir | oss:// |
| Hadoop S3 SDK | Kümeniz veya araçlarınız S3 uyumlu API'ler kullanır; Alibaba Cloud'a özel bağımlılık yoktur | s3a:// |
| JindoSDK | Jindo hızlandırması ile optimize edilmiş OSS performansı | oss:// |
Tüm adımlarda aynı SDK'yı kullanın.
Adım 1: Test verilerini hazırlama ve yükleme
Bir PageRank veri kümesi oluşturun ve bunu OSS paketinize yükleyin.
-
Aşağıdaki içerikle
generate_pagerank_dataset.shdosyasını oluşturun:#!/bin/bash # Check the number of arguments if [ "$#" -ne 2 ]; then echo "Usage: $0 M N" echo "M: Number of web pages" echo "N: Number of records to generate" exit 1 fi M=$1 N=$2 # Verify if M and N are positive integers if ! [[ "$M" =~ ^[0-9]+$ ]] || ! [[ "$N" =~ ^[0-9]+$ ]]; then echo "Both M and N must be positive integers." exit 1 fi # Generate dataset for ((i=1; i<=$N; i++)); do # Ensure the source and target pages are different while true; do src=$((RANDOM % M + 1)) dst=$((RANDOM % M + 1)) if [ "$src" -ne "$dst" ]; then echo "$src $dst" break fi done done -
Veri kümesini oluşturun:
M=100000 # The number of web pages N=10000000 # The number of records # Generate dataset randomly and save as pagerank_dataset.txt bash generate_pagerank_dataset.sh $M $N > pagerank_dataset.txt -
Veri kümesini OSS paketinizdeki
data/dizinine yükleyin:ossutil cp pagerank_dataset.txt oss://<BUCKET_NAME>/data/
Adım 2: Bir Spark konteyner görüntüsü oluşturma
OSS erişimi için gereken JAR bağımlılıklarını içeren bir konteyner görüntüsü oluşturun. Bkz. Görüntü oluşturmak için Container Registry Enterprise Edition örneği kullanma.
Örnek Dockerfile'lardaki temel Spark görüntüsü açık kaynak topluluğundan gelir. Gerektiğinde değiştirin ve SDK sürümünü Spark sürümünüzle eşleştirin.
Hadoop OSS SDK Kullan
Bu örnekte Spark 3.5.5 ve Hadoop OSS SDK 3.3.4 kullanılmaktadır:
ARG SPARK_IMAGE=spark:3.5.5
FROM ${SPARK_IMAGE}
# Add dependencies for Hadoop Aliyun OSS support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars
Hadoop S3 SDK Kullan
Bu örnekte Spark 3.5.5 ve Hadoop S3 SDK 3.3.4 kullanılmaktadır:
ARG SPARK_IMAGE=spark:3.5.5
FROM ${SPARK_IMAGE}
# Add dependencies for Hadoop AWS S3 support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.367/aws-java-sdk-bundle-1.12.367.jar ${SPARK_HOME}/jars
JindoSDK Kullan
Bu örnekte Spark 3.5.5 ve JindoSDK 6.8.0 kullanılmaktadır:
ARG SPARK_IMAGE=spark:3.5.5
FROM ${SPARK_IMAGE}
# Add dependencies for JindoSDK support
ADD --chown=spark:spark --chmod=644 https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/mvn-repo/com/aliyun/jindodata/jindo-core/6.8.0/jindo-core-6.8.0.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/mvn-repo/com/aliyun/jindodata/jindo-sdk/6.8.0/jindo-sdk-6.8.0.jar ${SPARK_HOME}/jars
Adım 3: OSS kimlik bilgilerini bir Kubernetes Secret içinde saklama
AccessKey ID ve AccessKey Secret bilgilerinizi SparkApplication manifest dosyasına gömmek yerine bir Kubernetes Secret içinde saklayın. Spark operatörü, Secret anahtarlarını sürücü ve yürütücü pod'larına ortam değişkenleri olarak enjekte eder.
Adım 4'tekihadoopConfalanlarıEnvironmentVariableCredentialsProviderkullanır; bu sağlayıcı söz konusu ortam değişkenlerini otomatik olarak okur. Kimlik bilgilerini YAML dosyalarına asla gömmeyin veya bunları kaynak denetimine kaydetmeyin.
Hadoop OSS SDK Kullan
-
spark-oss-secret.yamldosyasını oluşturun:apiVersion: v1 kind: Secret metadata: name: spark-oss-secret namespace: default stringData: # Replace <ACCESS_KEY_ID> with the AccessKey ID of your Alibaba Cloud account. OSS_ACCESS_KEY_ID: <ACCESS_KEY_ID> # Replace <ACCESS_KEY_SECRET> with the AccessKey Secret of your Alibaba Cloud account. OSS_ACCESS_KEY_SECRET: <ACCESS_KEY_SECRET> -
Secret'i uygulayın:
kubectl apply -f spark-oss-secret.yamlBeklenen çıktı:
secret/spark-oss-secret created
Hadoop S3 SDK Kullan
Hadoop S3 SDK, kimlik bilgilerini AWS_ACCESS_KEY_ID ve AWS_SECRET_ACCESS_KEY değerlerinden okur; bu nedenle anahtar adları OSS SDK'dan farklıdır.
-
spark-s3-secret.yamldosyasını oluşturun:apiVersion: v1 kind: Secret metadata: name: spark-s3-secret namespace: default stringData: # Replace <ACCESS_KEY_ID> with the AccessKey ID of your Alibaba Cloud account. AWS_ACCESS_KEY_ID: <ACCESS_KEY_ID> # Replace <ACCESS_KEY_SECRET> with the AccessKey Secret of your Alibaba Cloud account. AWS_SECRET_ACCESS_KEY: <ACCESS_KEY_SECRET> -
Secret'i uygulayın:
kubectl apply -f spark-s3-secret.yamlBeklenen çıktı:
secret/spark-s3-secret created
JindoSDK Kullan
-
spark-oss-secret.yamldosyasını oluşturun:apiVersion: v1 kind: Secret metadata: name: spark-oss-secret namespace: default stringData: # Replace <ACCESS_KEY_ID> with the AccessKey ID of your Alibaba Cloud account. OSS_ACCESS_KEY_ID: <ACCESS_KEY_ID> # Replace <ACCESS_KEY_SECRET> with the AccessKey Secret of your Alibaba Cloud account. OSS_ACCESS_KEY_SECRET: <ACCESS_KEY_SECRET> -
Secret'i uygulayın:
kubectl apply -f spark-oss-secret.yamlBeklenen çıktı:
secret/spark-oss-secret created
Adım 4: Spark işini gönderme
OSS veri kümenize karşı PageRank işini çalıştırmak için bir SparkApplication manifest dosyası oluşturun ve gönderin.
Hadoop OSS SDK Kullan
spark-pagerank.yaml dosyasını oluşturun. Hadoop-Aliyun modülü mevcut tüm OSS parametrelerini açıklar.
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pagerank
namespace: default
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with the Spark container image built in Step 2.
image: <SPARK_IMAGE>
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
mainClass: org.apache.spark.examples.SparkPageRank
arguments:
- oss://<OSS_BUCKET>/data/pagerank_dataset.txt # Specify the input test dataset. Replace <OSS_BUCKET> with your OSS bucket name.
- "10" # The number of iterations.
sparkVersion: 3.5.5
hadoopConf:
fs.AbstractFileSystem.oss.impl: org.apache.hadoop.fs.aliyun.oss.OSS
fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
# OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
# For example, the internal endpoint for the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com.
fs.oss.endpoint: <OSS_ENDPOINT>
fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
driver:
cores: 1
coreLimit: 1200m
memory: 512m
envFrom:
- secretRef:
name: spark-oss-secret # Specify the Secret used to access OSS.
serviceAccount: spark-operator-spark
executor:
instances: 2
cores: 1
coreLimit: "2"
memory: 8g
envFrom:
- secretRef:
name: spark-oss-secret # Specify the Secret used to access OSS.
restartPolicy:
type: Never
Hadoop S3 SDK Kullan
spark-pagerank.yaml dosyasını oluşturun. Hadoop-AWS modülü mevcut tüm S3 parametrelerini açıklar.
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pagerank
namespace: default
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with the Spark container image built in Step 2.
image: <SPARK_IMAGE>
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
mainClass: org.apache.spark.examples.SparkPageRank
arguments:
- s3a://<OSS_BUCKET>/data/pagerank_dataset.txt # Specify the input test dataset. Replace <OSS_BUCKET> with your OSS bucket name.
- "10" # The number of iterations.
sparkVersion: 3.5.5
hadoopConf:
fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
# OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
# For example, the internal endpoint for the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com.
fs.s3a.endpoint: <OSS_ENDPOINT>
# The region where the OSS endpoint is located. For example, cn-beijing for the China (Beijing) region.
fs.s3a.endpoint.region: <OSS_REGION>
driver:
cores: 1
coreLimit: 1200m
memory: 512m
envFrom:
- secretRef:
name: spark-s3-secret # Specify the Secret used to access OSS.
serviceAccount: spark-operator-spark
executor:
instances: 2
cores: 1
coreLimit: "2"
memory: 8g
envFrom:
- secretRef:
name: spark-s3-secret # Specify the Secret used to access OSS.
restartPolicy:
type: Never
JindoSDK Kullan
spark-pagerank.yaml dosyasını oluşturun:
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pagerank
namespace: default
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with the Spark container image built in Step 2.
image: <SPARK_IMAGE>
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
mainClass: org.apache.spark.examples.SparkPageRank
arguments:
- oss://<OSS_BUCKET>/data/pagerank_dataset.txt # Specify the input test dataset. Replace <OSS_BUCKET> with your OSS bucket name.
- "10" # The number of iterations.
sparkVersion: 3.5.5
hadoopConf:
fs.AbstractFileSystem.oss.impl: com.aliyun.jindodata.oss.JindoOSS
fs.oss.impl: com.aliyun.jindodata.oss.JindoOssFileSystem
# OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
# For example, the internal endpoint for the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com.
fs.oss.endpoint: <OSS_ENDPOINT>
fs.oss.credentials.provider: com.aliyun.jindodata.oss.auth.EnvironmentVariableCredentialsProvider
driver:
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
envFrom:
- secretRef:
name: spark-oss-secret # Specify the Secret used to access OSS.
executor:
instances: 2
cores: 1
coreLimit: "2"
memory: 8g
envFrom:
- secretRef:
name: spark-oss-secret # Specify the Secret used to access OSS.
restartPolicy:
type: Never
İşi gönderme ve doğrulama
Bu komutlar üç SDK seçeneği için de geçerlidir.
-
İşi gönderin:
kubectl apply -f spark-pagerank.yaml -
İş durumunu izleyin:
kubectl get sparkapplications spark-pagerankİş tamamlandığında beklenen çıktı:
NAME STATUS ATTEMPTS START FINISH AGE spark-pagerank COMPLETED 1 2024-10-09T12:54:25Z 2024-10-09T12:55:46Z 90s -
Sürücü pod'unun son 20 günlük satırını görüntüleyin:
kubectl logs spark-pagerank-driver --tail=20Hadoop OSS SDK — beklenen çıktı:
30024 has rank: 1.0709659078941967 . 21390 has rank: 0.9933356174074005 . 28500 has rank: 1.0404018494028928 . 2137 has rank: 0.9931000490520374 . 3406 has rank: 0.9562543137167121 . 20904 has rank: 0.8827028621652337 . 25604 has rank: 1.0270134041934191 . 24/10/09 12:48:36 INFO SparkUI: Stopped Spark web UI at http://spark-pagerank-dd0d4d927151c9d0-driver-svc.default.svc:4040 24/10/09 12:48:36 INFO KubernetesClusterSchedulerBackend: Shutting down all executors 24/10/09 12:48:36 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down 24/10/09 12:48:36 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed. 24/10/09 12:48:36 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 24/10/09 12:48:36 INFO MemoryStore: MemoryStore cleared 24/10/09 12:48:36 INFO BlockManager: BlockManager stopped 24/10/09 12:48:36 INFO BlockManagerMaster: BlockManagerMaster stopped 24/10/09 12:48:36 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 24/10/09 12:48:36 INFO SparkContext: Successfully stopped SparkContext 24/10/09 12:48:36 INFO ShutdownHookManager: Shutdown hook called 24/10/09 12:48:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-e8b8c2ab-c916-4f84-b60f-f54c0de3a7f0 24/10/09 12:48:36 INFO ShutdownHookManager: Deleting directory /var/data/spark-c5917d98-06fb-46fe-85bc-199b839cb885/spark-23e2c2ae-4754-43ae-854d-2752eb83b2c5Hadoop S3 SDK — beklenen çıktı:
3406 has rank: 0.9562543137167121 . 20904 has rank: 0.8827028621652337 . 25604 has rank: 1.0270134041934191 . 25/04/07 03:54:11 INFO SparkContext: SparkContext is stopping with exitCode 0. 25/04/07 03:54:11 INFO SparkUI: Stopped Spark web UI at http://spark-pagerank-0f7dec960e615617-driver-svc.spark.svc:4040 25/04/07 03:54:11 INFO KubernetesClusterSchedulerBackend: Shutting down all executors 25/04/07 03:54:11 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down 25/04/07 03:54:11 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed. 25/04/07 03:54:11 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 25/04/07 03:54:11 INFO MemoryStore: MemoryStore cleared 25/04/07 03:54:11 INFO BlockManager: BlockManager stopped 25/04/07 03:54:11 INFO BlockManagerMaster: BlockManagerMaster stopped 25/04/07 03:54:11 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 25/04/07 03:54:11 INFO SparkContext: Successfully stopped SparkContext 25/04/07 03:54:11 INFO ShutdownHookManager: Shutdown hook called 25/04/07 03:54:11 INFO ShutdownHookManager: Deleting directory /var/data/spark-20d425bb-f442-4b0a-83e2-5a0202959a54/spark-ff5bbf08-4343-4a7a-9ce0-3f7c127cf4a9 25/04/07 03:54:11 INFO ShutdownHookManager: Deleting directory /tmp/spark-a421839a-07af-49c0-b637-f15f76c3e752 25/04/07 03:54:11 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system... 25/04/07 03:54:11 INFO MetricsSystemImpl: s3a-file-system metrics system stopped. 25/04/07 03:54:11 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.JindoSDK — beklenen çıktı:
21390 has rank: 0.9933356174074005 . 28500 has rank: 1.0404018494028928 . 2137 has rank: 0.9931000490520374 . 3406 has rank: 0.9562543137167121 . 20904 has rank: 0.8827028621652337 . 25604 has rank: 1.0270134041934191 . 24/10/09 12:55:44 INFO SparkContext: SparkContext is stopping with exitCode 0. 24/10/09 12:55:44 INFO SparkUI: Stopped Spark web UI at http://spark-pagerank-6a5e3d9271584856-driver-svc.default.svc:4040 24/10/09 12:55:44 INFO KubernetesClusterSchedulerBackend: Shutting down all executors 24/10/09 12:55:44 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down 24/10/09 12:55:44 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed. 24/10/09 12:55:45 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 24/10/09 12:55:45 INFO MemoryStore: MemoryStore cleared 24/10/09 12:55:45 INFO BlockManager: BlockManager stopped 24/10/09 12:55:45 INFO BlockManagerMaster: BlockManagerMaster stopped 24/10/09 12:55:45 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 24/10/09 12:55:45 INFO SparkContext: Successfully stopped SparkContext 24/10/09 12:55:45 INFO ShutdownHookManager: Shutdown hook called 24/10/09 12:55:45 INFO ShutdownHookManager: Deleting directory /var/data/spark-87e8406e-06a7-4b4a-b18f-2193da299d35/spark-093a1b71-121a-4367-9d22-ad4e397c9815 24/10/09 12:55:45 INFO ShutdownHookManager: Deleting directory /tmp/spark-723e2039-a493-49e8-b86d-fff5fd1bb168
(İsteğe Bağlı) Adım 5: Temizleme
Artık gerekli olmadığında Spark işini ve Secret'i silin.
Spark işini silin:
kubectl delete -f spark-pagerank.yaml
Secret'i silin:
Hadoop OSS SDK veya JindoSDK:
kubectl delete -f spark-oss-secret.yaml
Hadoop S3 SDK:
kubectl delete -f spark-s3-secret.yaml