Task runtime resources in Dataphin refer to the CPU and memory allocations that tasks utilize during creation and execution. This topic outlines the customization of task runtime resources in Dataphin, enabling you to tailor resource allocations to your specific requirements.
Task runtime resource consumption
Resource consumption during task runtime is comprised of two elements: the container base and the task resources.
Container Base: These resources are preset by the system configuration and cannot be altered.
Task Resources: You have the ability to customize task resources. Dataphin enables resource configuration through the addition of annotation statements. For instructions, see task resource configuration description.
Task default resources
The default resource configurations for each task type are outlined in the table below.
Task type | Default task resources |
Shell/Python | CPU: 0.1 cores Memory: 256MB |
Offline pipeline/task sync | CPU: 0.5 cores Memory: 1GB |
SQL | No additional resources consumed. |
Spark (JAR/SQL) | CPU: 0.3 cores Memory: 4GB |
MapReduce | CPU: 0.2 cores Memory: 512MB |
Task resource configuration description
The range for custom resource configurations spans from 0.001CPU to 4.0CPU for CPU and 32MB to 16GB for memory. To customize task resource sizes, add annotation statements at the start of the task code:
@required_resource{required_memory=memory size;required_cpus=CPU size}Code Example:
!#/bin/bash
@required_resource{required_memory=2GB;required_cpus=1.0}
@resource_reference{"test-boot.jar"}
java -jar test-boot.jar -Xms 1024m -Xmx 1024m