AnalyticDB:Spark application performance diagnostics

Prerequisites

• You have created an AnalyticDB for MySQL Enterprise edition, Basic edition, or Data Lakehouse edition cluster. For more information, see Create a cluster.

• You have created a job-specific resource group with at least 8 ACU of reserved computing resources. For more information, see Create and manage resource groups.

• You have granted the AliyunADBDeveloperAccess permission to a RAM user. For more information, see RAM users and permissions .

• A database account is created for the AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster.

• AnalyticDB for MySQL is authorized to assume the AliyunADBSparkProcessingDataRole role to access other cloud resources.

Use cases

Use the Spark performance diagnostics feature in the following scenarios:

• Dataset performance analysis: When you process large datasets with Spark, the performance diagnostics tool can help you quickly identify bottlenecks, such as peak memory usage and spills, to improve data processing efficiency.

• Load balancing for large-scale applications: Spark applications running under high-concurrency workloads can experience performance issues such as data skew, long-tail tasks, and unbalanced loads. The performance diagnostics feature helps you quickly identify these issues and optimize your applications.

Limitations

• You can diagnose only Spark applications that have run successfully within the last 14 days.

• Only batch and streaming applications can be diagnosed.

Procedure

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the left-side navigation pane, choose Job Development>Spark JAR Development.

3. In the Applications section, find the target application, and in the Actions column, choose More > History.

4. In the Execution History section, find the target task and click Diagnose in the Actions column.

   Note

   After the diagnosis is complete, the Diagnostic Optimization Details panel automatically opens. If your Spark application has performance issues, you can optimize it based on the provided diagnostic information.