AnalyticDB:Access OSS

Prerequisites

Before you begin, ensure that you have:

• An AnalyticDB for MySQL Data Lakehouse Edition cluster created in the same region as your OSS bucket

• A job resource group created for the cluster. See Create a resource group

• A database account set up for the cluster:

  • Alibaba Cloud account users: a privileged account. See Create a privileged account

  • Resource Access Management (RAM) users: both a privileged account and a standard account, with the standard account associated with the RAM user. See Create a database account and Associate or disassociate a database account with or from a RAM user

• Authorization completed. See Perform authorization

Choose your scenario

Step 1: Prepare data

Upload both the input file and the PySpark script to your OSS bucket before submitting a job.

1. Create a text file named readme.txt with the following content, then upload it to your OSS bucket:

   AnalyticDB for MySQL
   Database service

   For upload instructions, see Upload objects.

2. Create a Python file named example.py that reads the first line of readme.txt, then upload it to the same OSS bucket:

   import sys
   from pyspark.sql import SparkSession
   
   # Initialize the Spark application
   spark = SparkSession.builder.appName('OSS Example').getOrCreate()
   
   # Read the text file — the path is passed in via args
   textFile = spark.sparkContext.textFile(sys.argv[1])
   
   # Print the total line count and the first line
   print("File total lines: " + str(textFile.count()))
   print("First line is: " + textFile.first())

Step 2: Submit a Spark job

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner, select a region. In the left-side navigation pane, click Clusters. On the Data Lakehouse Edition tab, find your cluster and click its ID.

2. In the left-side navigation pane, choose Job Development > Spark JAR Development.

3. In the upper part of the editor, select a job resource group and a Spark application type. In this example, the Batch type is selected.

4. Paste the job configuration JSON for your scenario and click Run Now.

   Access OSS data from the same account

   {
     "args": ["oss://testBucketName/data/readme.txt"],
     "name": "spark-oss-test",
     "file": "oss://testBucketName/data/example.py",
     "conf": {
       "spark.driver.resourceSpec": "small",
       "spark.executor.resourceSpec": "small",
       "spark.executor.instances": 1
     }
   }

   Access OSS data from a different account

   Add spark.adb.roleArn to the conf block. This parameter tells Spark which RAM role to assume when accessing the other account's OSS bucket.

   {
     "args": ["oss://testBucketName/data/readme.txt"],
     "name": "CrossAccount",
     "file": "oss://testBucketName/data/example.py",
     "conf": {
       "spark.adb.roleArn": "acs:ram::testAccountID:role/<testUserName>",
       "spark.driver.resourceSpec": "small",
       "spark.executor.resourceSpec": "small",
       "spark.executor.instances": 1
     }
   }

Parameters

Verify the result

After the job runs, click Log in the Actions column on the Applications tab of the Spark JAR Development page to view the output.

For more details on the log viewer, see Spark editor.

Related topics

• Overview of Spark application development

• Spark application configuration parameters