When all tasks share a single scheduling resource pool, low-priority jobs can monopolize resources and delay critical pipelines. Custom resource groups let you partition scheduling resources into isolated quotas, assign each quota to specific tenants and projects, and guarantee that high-priority tasks always have the capacity they need.
Limits
The custom resource group feature is available only to customers on the latest architecture. Contact the product operations team for access.
The default cluster supports up to 72 CPU cores for scheduling. To allocate additional CPU resources, purchase them separately. For the maximum allocable amount, see the scheduling resource dashboard.
Permissions
Super administrators, system administrators, and resource group owners can manage custom resource groups, including editing, enabling, disabling, and deleting them.
Use cases
Priority protection: Dedicating a resource group to critical pipelines prevents lower-priority jobs from consuming shared capacity and causing delays.
Project-level isolation: Assign separate resource groups to different business units so each project draws from its own quota. This supports per-unit billing and prevents any single project from impacting others.
Key concepts
Scheduling resources: Resources consumed when Dataphin schedules task instances. All scheduling resources for a tenant can be distributed across multiple resource groups, each linked to specific projects. Groups are fully isolated — activity in one group does not affect another.
Tenant default resource group: Each tenant has exactly one default resource group, created automatically when the tenant is created and cannot be deleted. When you create a custom resource group from the default cluster, the requested percentage is deducted from the tenant default resource group. Set a minimum reserved percentage for the tenant default resource group to ensure that if the assigned resource group is unavailable during scheduling, the tenant default resource group can serve as a fallback and task execution is not affected.
Custom resource group: Multiple custom resource groups can be created within each tenant. Each group has a defined purpose and resource quota. The two main scheduling scenarios are task daily scheduling and batch operations.
Task daily scheduling: Use a dedicated resource group for recurring schedules, ad hoc queries, and one-off executions. Associate the relevant projects with this group to control which tasks draw from its quota.
Batch operations: For data backfill and other one-time runs, temporarily assign a resource group to specify the capacity used for that batch job.
Resource group priority: The resource group specified in a task's scheduling configuration takes highest priority. If that group is disabled or deleted when the task instance is generated, the system falls back to the project default resource group. If the project default is also unavailable, the system falls back to the tenant's default resource group. The full fallback chain is:
Task-specified resource group > Project default resource group > Tenant default resource group
If the resource group assigned for task execution is disabled or deleted after the instance starts, the system automatically switches to the tenant's default resource group to complete execution.
Procedure overview
-
After deploying Dataphin, the system calculates the total available resources in your default cluster from the cluster's scheduling configuration and the purchase specifications:
Total available CPU is the lesser of the CPU available for scheduling per the cluster configuration and the CPU allocated in the purchased custom resource group configuration.
Total available memory is the memory available for scheduling within the cluster configuration.
If the custom resource group feature is enabled, set the maximum percentage of default cluster scheduling resources each tenant can use when creating the tenant. For details, see tenant settings.
-
Within each tenant, create multiple custom resource groups based on your resource management needs. For each group, specify the scheduling cluster and the percentage of cluster resources it can use.
When you create a custom resource group from the tenant default cluster, the requested percentage is automatically deducted from the tenant default resource group (default cluster).
When you create a resource group from a custom cluster, the requested percentage is automatically deducted from that custom cluster.
Resource group effective priority
During scheduling, Dataphin uses the resource group defined in the task's scheduling configuration. If that group is in an abnormal state or has been deleted, the task falls back to the project default resource group. If the project default is also unavailable, it falls back to the tenant's default resource group.
To apply emergency flow control, set a resource group's allocation to zero. Tasks assigned to that group enter a waiting for scheduling resources state until capacity becomes available.
Task resource group configuration
Both recurring and one-time tasks support scheduling resource group configuration.
Basic mode projects: Test runs, smoke tests, and recurring schedules all use the designated scheduling resource group.
Dev-Prod mode projects: Test runs, smoke tests, and development environment schedules use the project default resource group for the development project ; custom modifications are not permitted. Production environment recurring schedules use the resource group assigned to each task.
Resource group configuration entry
On the Dataphin home page, click Management Center in the top menu bar, then select System Settings.
-
Navigate to the Resource Settings page:
Click Resource Settings, then click Resource Group Configuration.
