The configuration parameters of AnalyticDB for MySQL Spark are similar to those of Apache Spark. This topic describes the configuration parameters of AnalyticDB for MySQL Spark that are different from those of Apache Spark.
Usage notes
Spark application configuration parameters are used to configure and adjust the behavior and performance of Spark applications. The format of Spark application configuration parameters varies based on Spark development tools.
Development tool | Configuration parameter format | Example |
SQL editor | set key=value; |
|
Spark JAR editor | "key": "value" |
|
Notebook editor | "key": "value" |
|
spark-submit | key=value |
|
Specify driver and executor resources
Parameter | Required | Description | Corresponding parameter in Apache Spark |
spark.driver.resourceSpec | Yes | The resource specifications of the Spark driver. Each type corresponds to distinct specifications. For more information, see the Type column in the "Spark resource specifications" table of this topic. Important If you submit Spark applications, you can use Apache Spark parameters and configure the parameters based on the values of cores and memory that are described in the "Spark resource specifications" table of this topic. Example: | spark.driver.cores and spark.driver.memory |
spark.executor.resourceSpec | Yes | The resource specifications of a Spark executor. Each type corresponds to distinct specifications. For more information, see the Type column in the "Spark resource specifications" table of this topic. Important If you submit Spark applications, you can use Apache Spark parameters and configure the parameters based on the values of cores and memory that are described in the "Spark resource specifications" table of this topic. Example: | spark.executor.cores and spark.executor.memory |
spark.adb.driverDiskSize | No | The size of additional disk storage that is mounted on the Spark driver to meet large disk storage requirements. By default, the additional disk storage is mounted on the /user_data_dir directory. Unit: GiB. Valid values: (0,100]. Example: spark.adb.driverDiskSize=50Gi. In this example, the additional disk storage that is mounted on the Spark driver is set to 50 GiB. | N/A |
spark.adb.executorDiskSize | No | The size of additional disk storage that is mounted on a Spark executor to meet the requirements of shuffle operations. By default, the additional disk storage is mounted on the /shuffle_volume directory. Unit: GiB. Valid values: (0,100]. Example: spark.adb.executorDiskSize=50Gi. In this example, the additional disk storage that is mounted on a Spark executor is set to 50 GiB. | N/A |
Spark resource specifications
You can use reserved resources or elastic resources to execute Spark jobs. If you use the on-demand elastic resources of a job resource group to execute Spark jobs, the system calculates the number of used AnalyticDB compute units (ACUs) based on the Spark resource specifications and the CPU-to-memory ratio by using the following formulas:
1:2 CPU-to-memory ratio: Number of used ACUs = Number of CPU cores × 0.8.
1:4 CPU-to-memory ratio: Number of used ACUs = Number of CPU cores × 1.
1:8 CPU-to-memory ratio: Number of used ACUs = Number of CPU cores × 1.25.
For information about the prices of on-demand elastic resources, see Pricing for Data Lakehouse Edition (V3.0).
Type | Specifications | Used ACUs | ||
CPU cores | Memory (GB) | Disk storage1 (GB) | ||
c.small | 1 | 2 | 20 | 0.8 |
small | 1 | 4 | 20 | 1 |
m.small | 1 | 8 | 20 | 1.5 |
c.medium | 2 | 4 | 20 | 1.6 |
medium | 2 | 8 | 20 | 2 |
m.medium | 2 | 16 | 20 | 3 |
c.large | 4 | 8 | 20 | 3.2 |
large | 4 | 16 | 20 | 4 |
m.large | 4 | 32 | 20 | 6 |
c.xlarge | 8 | 16 | 20 | 6.4 |
xlarge | 8 | 32 | 20 | 8 |
m.xlarge | 8 | 64 | 20 | 12 |
c.2xlarge | 16 | 32 | 20 | 12.8 |
2xlarge | 16 | 64 | 20 | 16 |
m.2xlarge | 16 | 128 | 20 | 24 |
1Disk storage: The system is expected to occupy 1% of the disk storage. The actual available disk storage may be less than 20 GB.
Specify priorities for Spark jobs
Parameter | Required | Default value | Description |
spark.adb.priority | No | NORMAL | The priority of a Spark job. If resources are insufficient to execute all Spark jobs that are submitted, the queued jobs that have higher priorities are first executed. Valid values:
Important We recommend that you set this parameter to HIGH for all Spark streaming jobs. |
Access metadata
Parameter | Required | Default value | Description |
spark.sql.catalogImplementation | No |
| The type of the metadata to be accessed. Valid values:
|
spark.sql.hive.metastore.version | No |
| The version of the metadata to be accessed. Valid values:
Note
|
Examples
Access the metadata in AnalyticDB for MySQL.
spark.sql.hive.metastore.version=adb;Access the metadata in the built-in Hive Metastore of Apache Spark.
spark.sql.catalogImplementation=hive; spark.sql.hive.metastore.version=2.1.3;Access the metadata in the temporary directory.
spark.sql.catalogImplementation=in-memory;
Configure the Spark UI
Parameter | Required | Default value | Description |
spark.app.log.rootPath | No |
| The path of AnalyticDB for MySQL Spark job logs and output data of the Linux OS. By default, the folder named Spark application ID stores the following data:
|
spark.adb.event.logUploadDuration | No | false | Specifies whether to record the amount of time that is consumed to upload event logs. |
spark.adb.buffer.maxNumEvents | No | 1000 | The maximum number of events that are cached by the driver. |
spark.adb.payload.maxNumEvents | No | 10000 | The maximum number of events that are uploaded to Object Storage Service (OSS) at a time. |
spark.adb.event.pollingIntervalSecs | No | 0.5 | The interval at which two batches of events are uploaded to OSS. Unit: seconds. For example, a value of 0.5 indicates that events are uploaded every 0.5 seconds. |
spark.adb.event.maxPollingIntervalSecs | No | 60 | The maximum retry interval that is required to upload events to OSS if an upload fails. Unit: seconds. The interval between a failed upload and an upload retry is within the range of the |
spark.adb.event.maxWaitOnEndSecs | No | 10 | The maximum wait time for events to be uploaded to OSS. Unit: seconds. The maximum wait time is the interval between the start and completion of an upload. If an upload is not complete within the maximum wait time, the upload is retried. |
spark.adb.event.waitForPendingPayloadsSleepIntervalSecs | No | 1 | The amount of time that is required to wait for retrying an upload that fails to complete within the |
spark.adb.eventLog.rolling.maxFileSize | No | 209715200 | The maximum file size of event logs in OSS. Unit: bytes. Event logs are stored in OSS as multiple files, such as Eventlog.0 and Eventlog.1. You can specify the file size. |
Grant permissions to RAM users
Parameter | Required | Default value | Description |
spark.adb.roleArn | No | N/A | The Alibaba Cloud Resource Name (ARN) that is specified in the Resource Access Management (RAM) console for the RAM role that submits Spark applications. For more information, see RAM role overview. If you submit Spark applications as a RAM user, you must specify this parameter. If you submit Spark applications with an Alibaba Cloud account, you do not need to specify this parameter. Note If you have granted permissions to a RAM user in the RAM console, you do not need to specify this parameter. For more information, see Perform authorization within an Alibaba Cloud account. |
Enable the built-in data source connectors
Parameter | Required | Default value | Description |
spark.adb.connectors | No | N/A | The names of the built-in connectors of AnalyticDB for MySQL Spark. Separate multiple names with commas (,). Valid values: oss, hudi, delta, adb, odps, and external_hive. |
spark.hadoop.io.compression.codec.snappy.native | No | false | Specifies whether a Snappy file is in the standard Snappy format. By default, Hadoop recognizes the Snappy files that are edited in Hadoop. If you set this parameter to true, the standard Snappy library is used for decompression. If you set this parameter to false, the default Snappy library of Hadoop is used for decompression. |
Enable VPC access and data source access
Parameter | Required | Default value | Description |
spark.adb.eni.enabled | No | false | Specifies whether to enable Elastic Network Interface (ENI). If you use external tables to access other external data sources, you must enable ENI. Valid values:
|
spark.adb.eni.vswitchId | No | N/A | The ID of the vSwitch that is associated with an ENI. If you connect to AnalyticDB for MySQL from an Elastic Compute Service (ECS) instance over a virtual private cloud (VPC), you must specify a vSwitch ID for the VPC. Note If you have enabled VPC access, you must set the spark.adb.eni.enabled parameter to true. |
spark.adb.eni.securityGroupId | No | N/A | The ID of the security group that is associated with an ENI. If you connect to AnalyticDB for MySQL from an ECS instance over a VPC, you must specify a security group ID. Note If you have enabled VPC access, you must set the spark.adb.eni.enabled parameter to true. |
spark.adb.eni.extraHosts | No | N/A | The mappings between IP addresses and hostnames. This parameter allows Spark to resolve the hostnames of data sources. If you want to access a self-managed Hive data source, you must specify this parameter. Note
|
spark.adb.eni.adbHostAlias.enabled | No | false | Specifies whether to automatically write the resolved domain name information that AnalyticDB for MySQL requires to a mapping table of domain names and IP addresses. Valid values:
If you use an ENI to read data from or write data to EMR Hive, you must set this parameter to true. |
Configure application retries
Parameter | Required | Default value | Description |
spark.adb.maxAttempts | No | 1 | The maximum number of attempts that are allowed to run an application. The default value is 1, which specifies that no retry attempts are allowed. If you set this parameter to 3 for a Spark application, the application is run up to three times within a sliding window. |
spark.adb.attemptFailuresValidityInterval | No | Integer.MAX | The duration of the sliding window within which the system attempts to run an application. Unit: seconds. For example, if you set this parameter to 6000 for a Spark application, the system counts the number of attempts within the last 6,000 seconds after the application fails. If the number of attempts is less than the value of the spark.adb.maxAttempts parameter, the system retries to run the application. |
Specify a running environment for Spark jobs
The following table describes the configuration parameters that are required when you use the virtual environments technology to package a Python environment and submit Spark jobs.
Parameter | Required | Default value | Description |
spark.pyspark.python | No | N/A | The path of the Python interpreter on your on-premises device. |
Configuration parameters that are not supported for AnalyticDB for MySQL
AnalyticDB for MySQL Spark does not support the following configuration parameters of Apache Spark. These configuration parameters do not take effect for AnalyticDB for MySQL Spark.
Useless options(these options will be ignored):
--deploy-mode
--master
--packages, please use `--jars` instead
--exclude-packages
--proxy-user
--repositories
--keytab
--principal
--queue
--total-executor-cores
--driver-library-path
--driver-class-path
--supervise
-S,--silent
-i <filename>