AnalyticDB:Access data in OSS

Prerequisites

• An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.

• An Object Storage Service (OSS) bucket is created in the same region as the AnalyticDB for MySQL cluster.

• A job resource group is created for the AnalyticDB for MySQL cluster.

• A database account is created for the AnalyticDB for MySQL cluster.

  • If you use an Alibaba Cloud account, you need to only create a privileged account.

  • If you use a Resource Access Management (RAM) user, you must create a privileged account and a standard account and associate the standard account with the RAM user.

• Authorization has been completed. For more information, see Quick authorization.

  Important

  Same-account access requires the AliyunADBSparkProcessingDataRole permission. For cross-account access, you must grant permissions to the other Alibaba Cloud account.

Step 1: Prepare the data

1. Prepare a text file and upload it to an OSS bucket. This topic uses a text file named readme.txt as an example. For more information, see Upload files.

   AnalyticDB for MySQL
   Database product

2. Write a Python script and upload it to the OSS bucket. This example uses a Python script named example.py, which reads the first line of the readme.txt file.

   import sys
   
   from pyspark.sql import SparkSession
   
   # Initialize Spark.
   spark = SparkSession.builder.appName('OSS Example').getOrCreate()
   # Read the specified file. The file path is passed as an argument from 'args'.
   textFile = spark.sparkContext.textFile(sys.argv[1])
   # Count and print the total number of lines in the file.
   print("File total lines: " + str(textFile.count()))
   # Print the first line of the file.
   print("First line is: " + textFile.first())
   

Step 2: Access the OSS data

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the left-side navigation pane, click Job Development > Spark JAR Development.

3. Above the editor, select a job resource group and a spark application type. This topic uses the Batch type as an example.

4. In the editor, enter the following Spark code to read the file from OSS and print its total line count and first line.

   Same-account access

   {
     "args": ["oss://testBucketName/data/readme.txt"],
     "name": "spark-oss-test",
     "file": "oss://testBucketName/data/example.py",
     "conf": {
       "spark.driver.resourceSpec": "small",
       "spark.executor.resourceSpec": "small",
       "spark.executor.instances": 1
     }
   }

   Parameters:

   Parameter	Description
   args	Arguments for the spark application. In this example, the OSS path of the text file is assigned to textFile.
   name	The name of the spark application.
   file	The storage path of the main file for the spark application. The main file can be a JAR package that contains the entry class or an executable Python script. Important The main file for a spark application can currently only be stored in OSS.
   spark.adb.roleArn	The RAM role for cross-account access to an external data source. You can specify multiple roles by separating them with commas (,). The format is acs:ram::<testAccountID>:role/<testUserName>. <testAccountID>: The ID of the Alibaba Cloud account to which the external data source belongs. <testUserName>: The name of the RAM role created for cross-account authorization. For more information, see Authorize accounts. Note This parameter is not required for same-account access to OSS.
   conf	The configuration parameters for the spark application. These are mostly consistent with standard Apache Spark parameters. The format is key: value. Use commas (,) to separate multiple parameters. For parameters that differ from Apache Spark or are specific to AnalyticDB for MySQL, see Spark application configuration parameters.

   Cross-account access

   {
     "args": ["oss://testBucketName/data/readme.txt"],
     "name": "CrossAccount",
     "file": "oss://testBucketName/data/example.py",
     "conf": {
       "spark.adb.roleArn": "acs:ram::testAccountID:role/<testUserName>",
       "spark.driver.resourceSpec": "small",
       "spark.executor.resourceSpec": "small",
       "spark.executor.instances": 1  
     }
   }

   Parameters:

   Parameter	Description
   args	Arguments for the spark application. In this example, the OSS path of the text file is assigned to textFile.
   name	The name of the spark application.
   file	The storage path of the main file for the spark application. The main file can be a JAR package that contains the entry class or an executable Python script. Important The main file for a spark application can currently only be stored in OSS.
   spark.adb.roleArn	The RAM role for cross-account access to an external data source. You can specify multiple roles by separating them with commas (,). The format is acs:ram::<testAccountID>:role/<testUserName>. <testAccountID>: The ID of the Alibaba Cloud account to which the external data source belongs. <testUserName>: The name of the RAM role created for cross-account authorization. For more information, see Authorize accounts. Note This parameter is not required for same-account access to OSS.
   conf	The configuration parameters for the spark application. These are mostly consistent with standard Apache Spark parameters. The format is key: value. Use commas (,) to separate multiple parameters. For parameters that differ from Apache Spark or are specific to AnalyticDB for MySQL, see Spark application configuration parameters.

5. Click Run Now.

   After the job is complete, you can view the output in the job log. To do so, go to the Applications on the Spark JAR Development page and click Logs for your job. For more information, see Spark development editor.

References

• For an overview of spark application development, see Introduction to Spark application development.

• For details about spark application configuration parameters, see Spark application configuration parameters.