Spark application performance diagnostics - AnalyticDB - Alibaba Cloud Documentation Center

AnalyticDB for MySQL Enterprise Edition, Basic Edition, and Data Lakehouse Edition (V3.0) provides the Spark application diagnostics feature. If a Spark application that you submit has performance issues, you can use the diagnostic information to quickly locate and analyze performance bottlenecks, optimize the Spark application, and improve problem-solving efficiency. This topic describes how to diagnose the performance of Spark applications and provides examples.

Prerequisites

You have created an AnalyticDB for MySQL Data Lakehouse Edition (V3.0) cluster. For more information, see Create a cluster.
You have created a job resource group with at least 8 ACUs of reserved computing resources. For more information, see Create and manage resource groups.
You have granted the AliyunADBDeveloperAccess permission to a Resource Access Management (RAM) user. For more information, see RAM users and permissions.
A database account is created for the AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster.
AnalyticDB for MySQL is authorized to assume the AliyunADBSparkProcessingDataRole role to access other cloud resources.

Use cases

This feature is primarily used in the following scenarios:

Dataset performance analysis: When processing large-scale data with Spark, performance analysis is essential. The diagnostics tool helps you quickly identify performance bottlenecks, such as memory spikes and disk spills, to improve data processing efficiency.
Load balancing for large-scale applications: When Spark applications run under high-concurrency workloads, performance issues such as data skew, long-tail tasks, and load imbalance can occur. Diagnosing the application helps you quickly identify these issues so you can optimize the application.

Limitations

You can diagnose only Spark applications that completed successfully within the last 14 days.
Only batch and streaming applications are supported.

Procedure

Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.
In the left-side navigation pane, choose Job Development>Spark JAR Development.
In the Workspaces section, find the target application and choose More > History in the Actions column.
In the Tuning History section, find the target task and click Diagnose in the Actions column.

Note
After the diagnosis is complete, the Diagnostic Optimization Details panel automatically opens. If the application has performance issues, use the information in this panel to optimize it.