Tüm Ürünler
Search
Belge Merkezi

Container Service for Kubernetes:Read and write OSS data in Spark jobs

Son Güncelleme:Oct 06, 2026

ACK kümelerinde Hadoop OSS SDK, Hadoop S3 SDK veya JindoSDK kullanarak yerleşik PageRank işiyle OSS verilerini okuyan ve yazan Spark işlerini çalıştırın.

Önkoşullar

Aşağıdakilere sahip olduğunuzdan emin olun:

Bir OSS entegrasyonu seçin

OSS erişimi için üç SDK mevcuttur. Devam etmeden önce birini seçin.

SDK Ne zaman kullanılır URI şeması
Hadoop OSS SDK Aliyun dosya sistemi aracılığıyla yerel OSS desteği; çoğu iş yükü için basittir oss://
Hadoop S3 SDK Kümeniz veya araçlarınız S3 uyumlu API'ler kullanır; Alibaba Cloud'a özel bağımlılık yoktur s3a://
JindoSDK Jindo hızlandırması ile optimize edilmiş OSS performansı oss://

Tüm adımlarda aynı SDK'yı kullanın.

Adım 1: Test verilerini hazırlama ve yükleme

Bir PageRank veri kümesi oluşturun ve bunu OSS paketinize yükleyin.

  1. Aşağıdaki içerikle generate_pagerank_dataset.sh dosyasını oluşturun:

    #!/bin/bash
    
    # Check the number of arguments
    if [ "$#" -ne 2 ]; then
        echo "Usage: $0 M N"
        echo "M: Number of web pages"
        echo "N: Number of records to generate"
        exit 1
    fi
    
    M=$1
    N=$2
    
    # Verify if M and N are positive integers
    if ! [[ "$M" =~ ^[0-9]+$ ]] || ! [[ "$N" =~ ^[0-9]+$ ]]; then
        echo "Both M and N must be positive integers."
        exit 1
    fi
    
    # Generate dataset
    for ((i=1; i<=$N; i++)); do
        # Ensure the source and target pages are different
        while true; do
            src=$((RANDOM % M + 1))
            dst=$((RANDOM % M + 1))
            if [ "$src" -ne "$dst" ]; then
                echo "$src $dst"
                break
            fi
        done
    done
  2. Veri kümesini oluşturun:

    M=100000    # The number of web pages
    
    N=10000000  # The number of records
    
    # Generate dataset randomly and save as pagerank_dataset.txt
    bash generate_pagerank_dataset.sh $M $N > pagerank_dataset.txt
  3. Veri kümesini OSS paketinizdeki data/ dizinine yükleyin:

    ossutil cp pagerank_dataset.txt oss://<BUCKET_NAME>/data/

Adım 2: Bir Spark konteyner görüntüsü oluşturma

OSS erişimi için gereken JAR bağımlılıklarını içeren bir konteyner görüntüsü oluşturun. Bkz. Görüntü oluşturmak için Container Registry Enterprise Edition örneği kullanma.

Örnek Dockerfile'lardaki temel Spark görüntüsü açık kaynak topluluğundan gelir. Gerektiğinde değiştirin ve SDK sürümünü Spark sürümünüzle eşleştirin.

Hadoop OSS SDK Kullan

Bu örnekte Spark 3.5.5 ve Hadoop OSS SDK 3.3.4 kullanılmaktadır:

ARG SPARK_IMAGE=spark:3.5.5

FROM ${SPARK_IMAGE}

# Add dependencies for Hadoop Aliyun OSS support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars

Hadoop S3 SDK Kullan

Bu örnekte Spark 3.5.5 ve Hadoop S3 SDK 3.3.4 kullanılmaktadır:

ARG SPARK_IMAGE=spark:3.5.5

FROM ${SPARK_IMAGE}

# Add dependencies for Hadoop AWS S3 support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.367/aws-java-sdk-bundle-1.12.367.jar ${SPARK_HOME}/jars

JindoSDK Kullan

Bu örnekte Spark 3.5.5 ve JindoSDK 6.8.0 kullanılmaktadır:

ARG SPARK_IMAGE=spark:3.5.5

FROM ${SPARK_IMAGE}

# Add dependencies for JindoSDK support
ADD --chown=spark:spark --chmod=644 https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/mvn-repo/com/aliyun/jindodata/jindo-core/6.8.0/jindo-core-6.8.0.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/mvn-repo/com/aliyun/jindodata/jindo-sdk/6.8.0/jindo-sdk-6.8.0.jar ${SPARK_HOME}/jars

Adım 3: OSS kimlik bilgilerini bir Kubernetes Secret içinde saklama

AccessKey ID ve AccessKey Secret bilgilerinizi SparkApplication manifest dosyasına gömmek yerine bir Kubernetes Secret içinde saklayın. Spark operatörü, Secret anahtarlarını sürücü ve yürütücü pod'larına ortam değişkenleri olarak enjekte eder.

Adım 4'teki hadoopConf alanları EnvironmentVariableCredentialsProvider kullanır; bu sağlayıcı söz konusu ortam değişkenlerini otomatik olarak okur. Kimlik bilgilerini YAML dosyalarına asla gömmeyin veya bunları kaynak denetimine kaydetmeyin.

Hadoop OSS SDK Kullan

  1. spark-oss-secret.yaml dosyasını oluşturun:

    apiVersion: v1
    kind: Secret
    metadata:
      name: spark-oss-secret
      namespace: default
    stringData:
      # Replace <ACCESS_KEY_ID> with the AccessKey ID of your Alibaba Cloud account.
      OSS_ACCESS_KEY_ID: <ACCESS_KEY_ID>
      # Replace <ACCESS_KEY_SECRET> with the AccessKey Secret of your Alibaba Cloud account.
      OSS_ACCESS_KEY_SECRET: <ACCESS_KEY_SECRET>
  2. Secret'i uygulayın:

    kubectl apply -f spark-oss-secret.yaml

    Beklenen çıktı:

    secret/spark-oss-secret created

Hadoop S3 SDK Kullan

Hadoop S3 SDK, kimlik bilgilerini AWS_ACCESS_KEY_ID ve AWS_SECRET_ACCESS_KEY değerlerinden okur; bu nedenle anahtar adları OSS SDK'dan farklıdır.

  1. spark-s3-secret.yaml dosyasını oluşturun:

    apiVersion: v1
    kind: Secret
    metadata:
      name: spark-s3-secret
      namespace: default
    stringData:
      # Replace <ACCESS_KEY_ID> with the AccessKey ID of your Alibaba Cloud account.
      AWS_ACCESS_KEY_ID: <ACCESS_KEY_ID>
      # Replace <ACCESS_KEY_SECRET> with the AccessKey Secret of your Alibaba Cloud account.
      AWS_SECRET_ACCESS_KEY: <ACCESS_KEY_SECRET>
  2. Secret'i uygulayın:

    kubectl apply -f spark-s3-secret.yaml

    Beklenen çıktı:

    secret/spark-s3-secret created

JindoSDK Kullan

  1. spark-oss-secret.yaml dosyasını oluşturun:

    apiVersion: v1
    kind: Secret
    metadata:
      name: spark-oss-secret
      namespace: default
    stringData:
      # Replace <ACCESS_KEY_ID> with the AccessKey ID of your Alibaba Cloud account.
      OSS_ACCESS_KEY_ID: <ACCESS_KEY_ID>
      # Replace <ACCESS_KEY_SECRET> with the AccessKey Secret of your Alibaba Cloud account.
      OSS_ACCESS_KEY_SECRET: <ACCESS_KEY_SECRET>
  2. Secret'i uygulayın:

    kubectl apply -f spark-oss-secret.yaml

    Beklenen çıktı:

    secret/spark-oss-secret created

Adım 4: Spark işini gönderme

OSS veri kümenize karşı PageRank işini çalıştırmak için bir SparkApplication manifest dosyası oluşturun ve gönderin.

Hadoop OSS SDK Kullan

spark-pagerank.yaml dosyasını oluşturun. Hadoop-Aliyun modülü mevcut tüm OSS parametrelerini açıklar.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pagerank
  namespace: default
spec:
  type: Scala
  mode: cluster
  # Replace <SPARK_IMAGE> with the Spark container image built in Step 2.
  image: <SPARK_IMAGE>
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
  mainClass: org.apache.spark.examples.SparkPageRank
  arguments:
  - oss://<OSS_BUCKET>/data/pagerank_dataset.txt           # Specify the input test dataset. Replace <OSS_BUCKET> with your OSS bucket name.
  - "10"                                                   # The number of iterations.
  sparkVersion: 3.5.5
  hadoopConf:
    fs.AbstractFileSystem.oss.impl: org.apache.hadoop.fs.aliyun.oss.OSS
    fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
    # OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
    # For example, the internal endpoint for the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com.
    fs.oss.endpoint: <OSS_ENDPOINT>
    fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    envFrom:
    - secretRef:
        name: spark-oss-secret               # Specify the Secret used to access OSS.
    serviceAccount: spark-operator-spark
  executor:
    instances: 2
    cores: 1
    coreLimit: "2"
    memory: 8g
    envFrom:
    - secretRef:
        name: spark-oss-secret               # Specify the Secret used to access OSS.
  restartPolicy:
    type: Never

Hadoop S3 SDK Kullan

spark-pagerank.yaml dosyasını oluşturun. Hadoop-AWS modülü mevcut tüm S3 parametrelerini açıklar.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pagerank
  namespace: default
spec:
  type: Scala
  mode: cluster
  # Replace <SPARK_IMAGE> with the Spark container image built in Step 2.
  image: <SPARK_IMAGE>
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
  mainClass: org.apache.spark.examples.SparkPageRank
  arguments:
  - s3a://<OSS_BUCKET>/data/pagerank_dataset.txt           # Specify the input test dataset. Replace <OSS_BUCKET> with your OSS bucket name.
  - "10"                                                   # The number of iterations.
  sparkVersion: 3.5.5
  hadoopConf:
    fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
    # OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
    # For example, the internal endpoint for the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com.
    fs.s3a.endpoint: <OSS_ENDPOINT>
    # The region where the OSS endpoint is located. For example, cn-beijing for the China (Beijing) region.
    fs.s3a.endpoint.region: <OSS_REGION>
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    envFrom:
    - secretRef:
        name: spark-s3-secret               # Specify the Secret used to access OSS.
    serviceAccount: spark-operator-spark
  executor:
    instances: 2
    cores: 1
    coreLimit: "2"
    memory: 8g
    envFrom:
    - secretRef:
        name: spark-s3-secret               # Specify the Secret used to access OSS.
  restartPolicy:
    type: Never

JindoSDK Kullan

spark-pagerank.yaml dosyasını oluşturun:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pagerank
  namespace: default
spec:
  type: Scala
  mode: cluster
  # Replace <SPARK_IMAGE> with the Spark container image built in Step 2.
  image: <SPARK_IMAGE>
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar
  mainClass: org.apache.spark.examples.SparkPageRank
  arguments:
  - oss://<OSS_BUCKET>/data/pagerank_dataset.txt    # Specify the input test dataset. Replace <OSS_BUCKET> with your OSS bucket name.
  - "10"                                            # The number of iterations.
  sparkVersion: 3.5.5
  hadoopConf:
    fs.AbstractFileSystem.oss.impl: com.aliyun.jindodata.oss.JindoOSS
    fs.oss.impl: com.aliyun.jindodata.oss.JindoOssFileSystem
    # OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
    # For example, the internal endpoint for the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com.
    fs.oss.endpoint: <OSS_ENDPOINT>
    fs.oss.credentials.provider: com.aliyun.jindodata.oss.auth.EnvironmentVariableCredentialsProvider
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    envFrom:
    - secretRef:
        name: spark-oss-secret                    # Specify the Secret used to access OSS.
  executor:
    instances: 2
    cores: 1
    coreLimit: "2"
    memory: 8g
    envFrom:
    - secretRef:
        name: spark-oss-secret                    # Specify the Secret used to access OSS.
  restartPolicy:
    type: Never

İşi gönderme ve doğrulama

Bu komutlar üç SDK seçeneği için de geçerlidir.

  1. İşi gönderin:

    kubectl apply -f spark-pagerank.yaml
  2. İş durumunu izleyin:

    kubectl get sparkapplications spark-pagerank

    İş tamamlandığında beklenen çıktı:

    NAME             STATUS      ATTEMPTS   START                  FINISH                 AGE
    spark-pagerank   COMPLETED   1          2024-10-09T12:54:25Z   2024-10-09T12:55:46Z   90s
  3. Sürücü pod'unun son 20 günlük satırını görüntüleyin:

    kubectl logs spark-pagerank-driver --tail=20

    Hadoop OSS SDK — beklenen çıktı:

    30024 has rank:  1.0709659078941967 .
    21390 has rank:  0.9933356174074005 .
    28500 has rank:  1.0404018494028928 .
    2137 has rank:  0.9931000490520374 .
    3406 has rank:  0.9562543137167121 .
    20904 has rank:  0.8827028621652337 .
    25604 has rank:  1.0270134041934191 .
    24/10/09 12:48:36 INFO SparkUI: Stopped Spark web UI at http://spark-pagerank-dd0d4d927151c9d0-driver-svc.default.svc:4040
    24/10/09 12:48:36 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
    24/10/09 12:48:36 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
    24/10/09 12:48:36 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
    24/10/09 12:48:36 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
    24/10/09 12:48:36 INFO MemoryStore: MemoryStore cleared
    24/10/09 12:48:36 INFO BlockManager: BlockManager stopped
    24/10/09 12:48:36 INFO BlockManagerMaster: BlockManagerMaster stopped
    24/10/09 12:48:36 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
    24/10/09 12:48:36 INFO SparkContext: Successfully stopped SparkContext
    24/10/09 12:48:36 INFO ShutdownHookManager: Shutdown hook called
    24/10/09 12:48:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-e8b8c2ab-c916-4f84-b60f-f54c0de3a7f0
    24/10/09 12:48:36 INFO ShutdownHookManager: Deleting directory /var/data/spark-c5917d98-06fb-46fe-85bc-199b839cb885/spark-23e2c2ae-4754-43ae-854d-2752eb83b2c5

    Hadoop S3 SDK — beklenen çıktı:

    3406 has rank:  0.9562543137167121 .
    20904 has rank:  0.8827028621652337 .
    25604 has rank:  1.0270134041934191 .
    25/04/07 03:54:11 INFO SparkContext: SparkContext is stopping with exitCode 0.
    25/04/07 03:54:11 INFO SparkUI: Stopped Spark web UI at http://spark-pagerank-0f7dec960e615617-driver-svc.spark.svc:4040
    25/04/07 03:54:11 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
    25/04/07 03:54:11 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
    25/04/07 03:54:11 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
    25/04/07 03:54:11 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
    25/04/07 03:54:11 INFO MemoryStore: MemoryStore cleared
    25/04/07 03:54:11 INFO BlockManager: BlockManager stopped
    25/04/07 03:54:11 INFO BlockManagerMaster: BlockManagerMaster stopped
    25/04/07 03:54:11 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
    25/04/07 03:54:11 INFO SparkContext: Successfully stopped SparkContext
    25/04/07 03:54:11 INFO ShutdownHookManager: Shutdown hook called
    25/04/07 03:54:11 INFO ShutdownHookManager: Deleting directory /var/data/spark-20d425bb-f442-4b0a-83e2-5a0202959a54/spark-ff5bbf08-4343-4a7a-9ce0-3f7c127cf4a9
    25/04/07 03:54:11 INFO ShutdownHookManager: Deleting directory /tmp/spark-a421839a-07af-49c0-b637-f15f76c3e752
    25/04/07 03:54:11 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
    25/04/07 03:54:11 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
    25/04/07 03:54:11 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.

    JindoSDK — beklenen çıktı:

    21390 has rank:  0.9933356174074005 .
    28500 has rank:  1.0404018494028928 .
    2137 has rank:  0.9931000490520374 .
    3406 has rank:  0.9562543137167121 .
    20904 has rank:  0.8827028621652337 .
    25604 has rank:  1.0270134041934191 .
    24/10/09 12:55:44 INFO SparkContext: SparkContext is stopping with exitCode 0.
    24/10/09 12:55:44 INFO SparkUI: Stopped Spark web UI at http://spark-pagerank-6a5e3d9271584856-driver-svc.default.svc:4040
    24/10/09 12:55:44 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
    24/10/09 12:55:44 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
    24/10/09 12:55:44 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
    24/10/09 12:55:45 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
    24/10/09 12:55:45 INFO MemoryStore: MemoryStore cleared
    24/10/09 12:55:45 INFO BlockManager: BlockManager stopped
    24/10/09 12:55:45 INFO BlockManagerMaster: BlockManagerMaster stopped
    24/10/09 12:55:45 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
    24/10/09 12:55:45 INFO SparkContext: Successfully stopped SparkContext
    24/10/09 12:55:45 INFO ShutdownHookManager: Shutdown hook called
    24/10/09 12:55:45 INFO ShutdownHookManager: Deleting directory /var/data/spark-87e8406e-06a7-4b4a-b18f-2193da299d35/spark-093a1b71-121a-4367-9d22-ad4e397c9815
    24/10/09 12:55:45 INFO ShutdownHookManager: Deleting directory /tmp/spark-723e2039-a493-49e8-b86d-fff5fd1bb168

(İsteğe Bağlı) Adım 5: Temizleme

Artık gerekli olmadığında Spark işini ve Secret'i silin.

Spark işini silin:

kubectl delete -f spark-pagerank.yaml

Secret'i silin:

Hadoop OSS SDK veya JindoSDK:

kubectl delete -f spark-oss-secret.yaml

Hadoop S3 SDK:

kubectl delete -f spark-s3-secret.yaml

Sonraki adımlar