Dataphin offers a resource group management feature that allows for the categorization of resources into distinct resource groups, each with its own separate quota. This enables the allocation of specific resource groups to tasks under various tenants and projects, ensuring that essential tasks have dedicated resources and enhancing overall resource utilization.
Limits
The custom resource group feature is currently available only to customers on the latest architecture. For more information, please contact the product operations team.
The default cluster can manage up to 72 CPU cores for scheduling. To control additional resources, you may purchase them separately. For the maximum allocable resource amount, please refer to the scheduling resource dashboard.
Permission description
Enables super administrators, system administrators, and resource group owners to manage custom resource groups, including editing, enabling, disabling, and deleting them.
Scenarios
Resource Group Management: This ensures that high-priority core tasks are executed first, preventing low-priority tasks from monopolizing scheduling resources and delaying the execution of more critical tasks.
Allocate on Demand: Resource groups facilitate the allocation of resources by project, ensuring that resources utilized by various business departments remain independent. This arrangement supports application scenarios like on-demand billing for individual business units.
Terminology
Scheduling resources: The term 'scheduling resources' denotes the resources utilized during the scheduling of task-generated instances. This feature allows for the distribution of all available scheduling resources for the current tenant across various resource groups, linking them to corresponding projects. Resource isolation between groups ensures that they operate independently without mutual interference.
Tenant default resource group: Each tenant has only one, which is automatically generated by the system when the tenant is created and cannot be deleted. When creating a custom resource group from the default cluster, the corresponding resources are automatically allocated from the tenant default resource group. To ensure that all tasks have available scheduling resources, it is necessary to set the minimum resource percentage reserved for the tenant default resource group. This ensures that if the specified resource group status is unavailable during scheduling, the tenant default resource group can be used as a fallback to avoid affecting task execution.
Custom resource group: Within each tenant, multiple custom resource groups can be established. These groups allow for the setting of specific application scenarios, enhancing control over resource usage. Notable application scenarios include task daily scheduling and batch operations.
Task Daily Scheduling: Custom resource groups are ideal for scenarios including ad hoc queries, temporary executions, and recurring schedules. Configuring the projects associated with the resource group is essential.
Batch operations: For single-time scheduling during data backfill tasks, resource groups can be designated temporarily to specify the resource group utilized for batch operations.
Resource group priority: The resource group designated for each task takes precedence. Should the custom resource group selected at the time of instance generation become disabled or deleted, the instance will default to the project's default resource group. In the event that this default resource group is also unavailable, the system will default to the tenant's default resource group. The hierarchy of resource group usage is as follows: Task-specified resource group > Project default resource group >Tenant default resource group. Moreover, if the resource group assigned for the scheduling of task execution is disabled or deleted, the system will automatically revert to the tenant's default resource group to carry out the execution.
Procedure overview
After deploying Dataphin, the system calculates the total available resources in your default cluster based on the cluster's scheduling resources and the resources specified in the purchase information. This includes:
Total available CPU is the lesser of the CPU resources available for scheduling as per the cluster configuration and the CPU resources designated for the purchased custom resource group configuration.
Total available memory refers to the memory resources that are available for scheduling within the cluster configuration.
If the custom resource group feature is enabled, when creating a tenant, you must allocate the maximum percentage of default cluster scheduling resources that each tenant can use. For details, see tenant settings.
Within each tenant, you can establish multiple custom resource groups based on resource management needs and specify the scheduling cluster and the percentage of available cluster resources each group will use.
When creating a custom resource group from the tenant default cluster, the required resource percentage will be automatically deducted from the tenant default resource group (default cluster).
When creating a resource group from a custom cluster, the required resource percentage will be automatically deducted from the custom cluster.
Resource group effective priority
During task scheduling, the system initially utilizes the resource group defined in the task's scheduling configuration. Should this resource group be in an abnormal state or deleted, the task will default to the project's default resource group. In the event that the project's default resource group is also inaccessible, the system will default to the tenant's default resource group. To implement emergency flow control, you may set the resource allocation for a specific resource group to zero. Consequently, the task will remain in a waiting for scheduling resources state until resources become available in the resource group.
Task resource group configuration description
Both recurring and one-time tasks allow for the configuration of scheduling resource groups. In Basic mode projects, task test runs, smoke tests, and recurring schedules utilize the designated scheduling resource group. For Dev-Prod mode projects, task test runs, smoke tests, and scheduling within the development environment adhere to the default project resource group set for the development project; custom modifications are not permitted. Conversely, production environment recurring schedules employ the resource group assigned to each specific task.
Resource group configuration entry
On the Dataphin home page, click Management Center in the top menu bar, then select System Settings.
To access the Resource Settings page, follow the steps below:
Click Resource Settings followed by Resource Group Configuration.
