This topic describes how to configure resources for Ververica Flink stream-batch integrated tasks in real-time mode. You can configure the clusters and engine versions for production and development environments. You can also configure the degree of parallelism, and the CPU and memory for JobManagers and TaskManagers.
Procedure
On the Dataphin home page, go to Develop > Data Development from the top menu bar.
From the top menu bar, select a Project. If you are in Dev-Prod mode, you must also select an Environment.
In the navigation pane on the left, choose Data Processing > Script Task. In the list of compute tasks on the right, click the target compute task.
In the right-side sidebar, click Configuration. In the panel that appears, click the Real-time Mode tab.
In the Resource Configuration section on the real-time mode configuration panel, you can configure the Resource Configuration settings.
Parameter
Description
Production Environment Cluster
The cluster for the production environment that you selected when you created the task is used by default. You can switch the cluster here.
ImportantAfter you switch the cluster, task execution may vary after submission. Proceed with caution.
Production Environment Engine Version
The engine version for the production environment that you selected when you created the task is used by default. You can switch the engine version here.
Development Environment Cluster and Engine Version
You can use the default system configuration or specify a custom configuration.
System default configuration: This is the default option. It uses the same cluster and engine version as the production environment.
Custom configuration: You can select the cluster and engine version for task execution in the development environment.
NoteIf your project is in Basic mode, you do not need to configure this item.
Parallelism
The default value is 4. You can enter any integer greater than 0. A value of -1 indicates automatic inference.
Number of Task Managers
The default value is 2. You can enter any integer greater than 0.
Task Manager CPUs
The default value is 1. You can enter any number greater than 0, such as 1 or 10.5.
Task Manager Memory
The default value is 8 Gi. Use Gi or Mi as the unit. You can enter a number in bytes, or a number with a memory unit, such as 1024000, 1024 Mi, or 1.5 Gi.
Job Manager CPUs
The amount of CPU resources that the task occupies in the control unit of the Flink cluster. The default value is 1. You can enter any number greater than 0, such as 1 or 10.5.
Job Manager Memory
The amount of memory resources that the task occupies in the control unit of the Flink cluster. The default value is 4 Gi. Use Gi or Mi as the unit. You can enter a number in bytes, or a number with a memory unit, such as 1024000, 1024 Mi, or 1.5 Gi.
Resource Configuration Type
System-recommended configuration
This is the intelligent mode (BETA) for fine-grained resource configuration in real-time computing Flink. Intelligent mode is an upgraded version of expert mode. In intelligent mode, the job uses the resource configurations of expert mode and enables the AutoPilot automatic tuning feature.
In auto configuration mode, you do not need to configure related resources. Autopilot automatically generates resource configurations for the job and adjusts the resource configurations based on the status of the job. This optimizes resource utilization of the job without affecting the health of the job. For more information, see Configure Autopilot.
Custom configuration
This is the expert mode (BETA) for fine-grained resource configuration in real-time computing Flink. This is a new resource configuration mode introduced by fully managed Flink. It supports fine-grained control over job resources to meet throughput requirements.
The system automatically runs the job in Native K8s mode based on the resource requirements that you configure, and automatically determines the specifications and number of Task Managers (TMs) based on the Slot specifications and job concurrency. For more information about configuring fine-grained resources, see Configure job deployment information.
To customize Flink SQL task resources in Dataphin, see Customize resources for Ververica Flink real-time tasks.
NoteFlink DataStream tasks do not support the Resource Configuration Type configuration.
Resource Configuration Records
When the Resource Configuration information changes, you can click Save Current Configuration as New Record. For a saved resource configuration record, you can View Details, Start, or Delete it.
NoteFlink DataStream tasks do not support Resource Configuration Records.
Click OK to save the configuration.