View the schedule resource dashboard - Dataphin - Alibaba Cloud Documentation Center

The schedule resource dashboard displays the usage and allocation of configured schedule resources in a Dataphin cluster. You can view and analyze the global resource configuration and individual task resource allocation. This helps improve resource utilization, reduce costs, and lower the risk of task backlogs due to resource shortages, which enhances platform stability.

Prerequisites

You must contact the Dataphin deployment team to deploy Prometheus monitoring and collect monitoring data before you can use the schedule resource dashboard.

The schedule resource dashboard switch must be turned on in Management Center > Resource Settings > Resource Monitoring for the metadata warehouse tenant. For more information, see Resource Monitoring.

Permission description

Only super administrators, system administrators, and custom global roles with the Development - Modify Default Cluster for Resource Dashboard permission can change the default cluster displayed on the schedule resource dashboard.

Feature description

The schedule resource dashboard analyzes Dataphin cluster schedule resources from two perspectives: resource allocation and resource consumption, with a focus on CPU and memory metrics. The dashboard also lists tasks that are recommended for optimization. These tasks may have low resource utilization, which wastes resources, or high utilization, which creates a risk of memory overflow. Identifying and resolving these issues promptly enhances platform stability and reduces costs.
The resource dashboard consists of three modules: resource allocation, resource consumption, and recommended optimization tasks. You can view the latest snapshot values to quickly identify and resolve blocking tasks that consume excessive resources and cause task backlogs. You can also analyze trends over time to identify optimization opportunities and improve resource utilization.

Data statistics frequency

Data is collected once per minute. For allocated resources, the dashboard displays a snapshot of the current value. For consumed resources, it displays the peak value from the past minute. The data on the page updates each time you refresh it.

Schedule resource dashboard entry

On the Dataphin home page, choose Development > Task O&M from the top menu bar.
In the navigation pane on the left, choose Overview > Schedule Resource Dashboard to open the Schedule Resource Dashboard page.

Tenant resource dashboard page introduction

Note

Dataphin instances that use a Mesos architecture do not support the tenant resource dashboard.

Section	Description
①Source Cluster	The default schedule cluster is selected by default. You can switch to any Normal status cluster that has metric collection enabled. You can also change the default display cluster for the schedule resource dashboard.
②Resource Overview	View and analyze the resources allocated by the selected source cluster. This includes the CPU Allocation Value and Memory Allocation Value. CPU Allocation Value: When the source cluster is the default cluster: Displays the current tenant's CPU allocation value (c), the available range set for the current tenant (upper and lower limits). When the source cluster is a registered schedule cluster: Displays the sum of allocated CPU values for all resource groups of the selected source cluster in the current tenant. This value is the same as the allocation value shown in the Resource Consumption section when All resource groups is selected. It also displays the available range. The upper limit of the available range is the total resources of the selected source cluster. The lower limit is the sum of the available lower limits of all resource groups of the selected source cluster in the current tenant. Memory Allocation Value: When the source cluster is the default cluster: Displays the current tenant's memory allocation value (G), the available range set for the current tenant (upper and lower limits). When the source cluster is a registered schedule cluster: Displays the sum of allocated memory values for all resource groups of the selected source cluster in the current tenant. This value is the same as the allocation value shown in the Resource Consumption section when All resource groups is selected. It also displays the available range. The upper limit of the available range is the total resources of the selected source cluster. The lower limit is the sum of the available lower limits of all resource groups of the selected source cluster in the current tenant. Note For Dataphin instances with a K8s architecture that do not support custom resource groups, only CPU and memory allocation values are displayed.
③Resource Consumption	Displays the CPU and memory consumption for the current tenant within a specified time and for a specific resource group. Time filter: The default selection is 3h. You can select the last 3h, 6h, 12h, 24h, 3d, 7d, or a custom time range. The span for a custom time range is from 10 minutes to 15 days (`[10 minutes, 15 days]`). Resource group filter: The default selection is All Resource Groups. You can select resource groups from the selected source cluster. Resource groups from the default schedule cluster are supported. Resource groups from a registered schedule cluster are not supported. CPU Allocation Value/Memory Allocation Value: The total amount of CPU or memory allocated to tasks at each statistical time point. The graph marks the maximum and minimum values within the selected time range as Max and Min. Available Lower Limit/Available Lower Limit: Displays the available upper and lower limits of CPU or memory for the current tenant or selected resource group, based on the selection in the resource group filter. Total Running Instances: The total number of instances in the running state at each statistical time point. This includes recurring instances, data backfill instances, and one-time instances. Total Waiting Instances: The number of instances waiting for schedule resources at each statistical time point. Note For Dataphin instances with a K8s architecture that do not support custom resource groups, resource group filtering is not supported, and the available upper and lower limits are not displayed in the graph.
④Recommended Optimization Tasks	Displays recommended optimization tasks for the current tenant by CPU and Memory dimensions. It only includes recurring tasks in the production environment that use resources from the selected schedule cluster. SQL and other shared container tasks are not included. The task list includes the following parameters: Task Name: Displays the task name and task ID. Click the task name to go to the Task O&M > Recurring Tasks page. Operation Owner: The O&M owner of the task. Project: The project to which the task belongs. Last Allocated CPU/Last Allocated Memory: The total amount of resources allocated to this task during its most recent execution. Last Consumed CPU/Last Consumed Memory: The total amount of resources consumed during the most recent execution of the task. Last Consumption Rate: The resource consumption rate during the most recent execution of the task (`Resource consumption rate = Total resources consumed during task execution / Total resources allocated to the task during execution`). Last Runtime: The runtime of the most recent execution of the task. Last 7 Consumption Values: The resource consumption values for the last seven runs of the recurring instances and data backfill instances for this task. Average Runtime Of Last 7 Runs: The average runtime of the last seven runs of the recurring instances and data backfill instances for this task. Actions: In the Actions column, perform the View Resource Details and Resource Configuration operations for the task. View Resource Details: Click the icon to open the Task Resource Details dialog box. You can view the task's Basic Information, Resource Details (details of the last run or trend of the last seven runs), and Resource Allocation Consumption Trend. Resource Configuration: Click the icon to go to the development page for the task. You can filter tasks in the upper-right corner of the list. The supported filter options include the following: Last Consumption Rate >> Lower Than >> 25%/50%/75% Last Consumption Rate >> Higher Than >> 50%/75%/90% Last Allocated CPU/Last Allocated Memory >> TOP20/TOP50/TOP100 Last Consumed CPU/Last Consumed Memory >> TOP20/TOP50/TOP100 Click Export in the upper-right corner of the list to export the task list with the current filters to an Excel file. Each value in the Last 7 Consumption Values column is separated by a comma (,) and placed in the same cell.

Recommendations for optimization tasks

If a task's allocated resources consistently exceed its consumed resources and the task has a long runtime, monitor it closely. This helps prevent the task from affecting ad hoc queries and other tasks, which could impact downstream services.
If a task's allocated resources consistently exceed its consumed resources but the task has a short runtime, consider reallocating the excess resources to other tasks that require them.
If a task's consumed resources consistently approach its allocated resources, monitor it to prevent delays or failures caused by insufficient resources.
If a task's memory usage reaches 100%, monitor it closely. You should increase its memory allocation to prevent failures caused by memory overflow, which could affect data output.

Cluster resource dashboard page introduction

Section	Description
①Source Cluster	The default schedule cluster is selected by default. You can switch to any Normal status cluster that has metric collection enabled. You can also change the default display cluster for the schedule resource dashboard.
②Resource Overview	View and analyze the resources allocated by the selected source cluster. This includes the CPU Allocation Value and Memory Allocation Value. CPU Allocation Value: Displays the CPU allocation value (c), available amount (c), and allocation rate (%) of the default schedule cluster. Memory Allocation Value: Displays the memory allocation value (G), available amount (G), and allocation rate (%) of the default schedule cluster.
③Resource Consumption	Displays the CPU and memory consumption for the default schedule cluster. Time filter: The default selection is 3h. You can select the last 3h, 6h, 12h, 24h, 3d, or 7d. CPU Allocation Value/Memory Allocation Value: The total amount of CPU or memory allocated to tasks at each statistical time point. The graph marks the maximum and minimum values within the selected time range as Max and Min. CPU Consumption Value/Memory Consumption Peak: The total amount of CPU or memory consumed by task execution at each statistical time point. Total Running Instances: The total number of instances in the running state at each statistical time point. This includes recurring instances, data backfill instances, and one-time instances.
④Recommended Optimization Tasks	When the consumption value is very different from the allocation value, use the optimization task list to find tasks with low consumption rates. Adjust their resource configurations to improve overall resource utilization. For more information, see Recommended optimization tasks and Recommendations for optimization tasks. Note Only Dataphin instances with a Mesos architecture support recommended optimization tasks.