AnalyticDB:Access DLF metadata by using Spark SQL

Prerequisites

• An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.

• A job resource group is created for the AnalyticDB for MySQL cluster.

• A database account is created for the AnalyticDB for MySQL cluster.

  • If you use an Alibaba Cloud account, you need to only create a privileged account.

  • If you use a Resource Access Management (RAM) user, you must create a privileged account and a standard account and associate the standard account with the RAM user.

• AnalyticDB for MySQL is authorized to assume the AliyunADBSparkProcessingDataRole role to access other cloud resources.

• DLF is activated, and a catalog exists in the same region as the AnalyticDB for MySQL cluster. For more information, see Create a catalog.

Procedure

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the left-side navigation pane, choose Job Development > SQL Development.

3. In the SQL Console window, select the Spark engine and a Job Resource Group.

4. Write a Spark SQL job.

   SET spark.hadoop.dlf.catalog.accessKeyId=LTAI5tMnzDS5EyqqMsT****;
   SET spark.hadoop.dlf.catalog.accessKeySecret=A2kHFzEgFidOKrDKqAbJIPt8****;
   SET spark.hadoop.dlf.catalog.id=default;
   SET spark.sql.hive.metastore.version=dlf;
   SHOW DATABASES;

   Parameters:

   Parameter	Required	Description
   spark.hadoop.dlf.catalog.accessKeyId	Yes	The AccessKey ID of an Alibaba Cloud account or a DLF that is authorized to access DLF. For information about how to obtain an AccessKey ID and an AccessKey Secret, see Accounts and permissions.
   spark.hadoop.dlf.catalog.accessKeySecret	Yes	The AccessKey Secret of an Alibaba Cloud account or a DLF that is authorized to access DLF. For information about how to obtain an AccessKey ID and an AccessKey Secret, see Accounts and permissions.
   spark.hadoop.dlf.catalog.id	Yes	The ID of the catalog. To query the catalog ID, see View a data catalog.
   spark.sql.hive.metastore.version	Yes	Specifies the version of the metastore service. Set this parameter to dlf.

   For information about other parameters, see Spark application configuration parameters.

5. Click Run Now. After the job is complete, you can view the output in Spark JAR Development under the Applications tab on the Logs page. For more information, see Spark development editor.