Dataphin:View the schedule resource dashboard

Prerequisites

• Ensure the Dataphin deployment team has set up Prometheus monitoring to collect the necessary data for the schedule resource dashboard.

• Verify that the schedule resource dashboard feature is enabled in Management Center > Resource Settings within the metadata warehouse tenant. For more information, see Resource Settings.

Feature description

• The schedule resource dashboard in Dataphin offers insights into the cluster's resource management from two key aspects: Resource Allocation and Resource Consumption, with a focus on CPU and memory metrics. It also highlights tasks with suboptimal resource usage-either underutilized, leading to waste, or overutilized, risking memory overflow. This optimization task list enables prompt issue identification and resolution, thereby bolstering platform stability and optimizing resource expenditures.

• Comprising three modules-Resource Allocation, Resource Consumption, and Suggested Optimization Tasks-the dashboard enables quick identification of blocking tasks that consume excessive resources, potentially causing task backlog. By examining snapshot values and trend analysis, you can assess optimization opportunities to enhance resource utilization.

Data statistics scope

The data statistics for the schedule resource dashboard are collected as follows:

• Data Statistics Frequency: Statistics are gathered every minute, capturing the current snapshot for allocated resources and the peak value within the past minute for consumed resources. A page refresh triggers an update of the page data.

• Data Statistics Scope:

  • Global Resource Consumption and Global Resource Allocation: These statistics encompass the allocation and consumption values of all task instances across all operational environments under the current tenant.

  • Suggested Optimization Tasks: Statistics are limited to auto-triggered nodes (Basic and Prod projects) running in dedicated containers within the production environment, excluding SQL tasks and code template tasks in shared containers.

Schedule resource dashboard entry

1. On the Dataphin home page, single click the top menu bar Development > Task Operation And Maintenance.

2. In the Operation Center, select Schedule Resource Dashboard from the left-side navigation pane to access the Schedule Resource Dashboard page.

   

Global resource allocation

Global resource allocation illustrates the distribution of CPU and memory relative to total resources at the current statistical time point and provides a historical trend chart of resource allocation.

Global resource allocation Metric explanation

The metrics for global resource allocation are detailed below.

The resource allocation trend chart allows you to select time frames such as the last 3, 6, 12, 24 hours, 3 days, or 7 days to observe the allocation trends. The horizontal axis is segmented according to the chosen time frame, displaying corresponding time points, while the vertical axis represents the resource allocation rate, marked at intervals of 0%, 20%, 40%, 60%, 80%, and 100%. Hovering over a time point reveals the CPU allocation value, CPU allocation rate, memory allocation value, and memory allocation rate at that moment.

Global resource allocation Optimization suggestions

Pay particular attention to the CPU Allocation Rate and Memory Allocation Rate. The following optimization suggestions are provided:

• A consistently low resource allocation rate indicates underutilization, leading to potential waste. Consider downsizing the total resources to save on costs.

• A consistently high allocation rate may result in task delays due to resource contention or failures from memory shortages. Increasing the total resources may be necessary.

Global resource consumption

Global resource consumption offers a comparative view of the actual resources consumed by global tasks against the pre-allocated resources, highlighting fluctuation trends.

When there is a significant discrepancy between the consumption and allocation values, it is advisable to examine the optimization task list for tasks with low consumption rates and adjust their resource allocation to enhance overall resource utilization. The following are detailed explanations:

• If there is a notable difference between the overall resource consumption and allocation values over a period, consider reviewing the suggested optimization tasks and potentially reducing the allocation for some tasks to improve resource utilization.

• If the overall consumption and allocation values are closely aligned over a period, the resource allocation is generally appropriate. However, consider increasing the allocation for certain core tasks to balance efficiency and stability requirements based on their historical performance.

Suggested optimization tasks

Suggested optimization tasks highlight tasks with resource consumption rates that exceed or fall below certain thresholds. This allows for flexible adjustment of resource allocation to ensure tasks have adequate resources for stable operation, preventing under-allocation that could disrupt normal scheduling or over-allocation that leads to resource waste.

Optimization tasks Metric explanation

Filter Suggested Optimization Tasks by project, task type, and consumption rate threshold. The metrics for the list items are as follows:

Optimization tasks Focus suggestions

• If a task consistently shows a resource allocation value significantly higher than its consumption value and has a long runtime, it should be prioritized for review to prevent it from impacting the execution of ad hoc queries and downstream business processes.

• If a task's resource allocation value is significantly higher than its consumption value over a period, but the runtime is short, consider a moderate adjustment of resources to reallocate the surplus to other tasks that may be resource-constrained.

• If a task's resource consumption value is close to its allocation value over time, monitor it to prevent delays or failures due to insufficient resources.

• If a task's memory consumption rate reaches 100%, it is critical to address this by increasing memory allocation to prevent failures due to memory overflow, which could affect data output.

Task resource details

Follow the steps below to view detailed task resource information. Task resource details provide insights into the most recent run and the trend of the last 7 runs, aiding in the optimization of resource allocation.