Configure resources for Ververica Flink stream-batch integrated tasks in real-time mode, including clusters, engine versions, parallelism, and CPU and memory for JobManagers and TaskManagers.
Procedure
-
On the Dataphin home page, go to Develop > Data Development from the top menu bar.
-
From the top menu bar, select a Project. If you are in Dev-Prod mode, you must also select an Environment.
-
In the navigation pane on the left, choose Data Processing > Script Task. In the list of compute tasks on the right, click the target compute task.
-
In the right-side sidebar, click Configuration. In the panel that appears, click the Real-time Mode tab.
-
In the Resource Configuration section on the real-time mode configuration panel, configure the following settings.
Parameter
Description
Production Environment Cluster
Defaults to the cluster selected when you created the task. You can switch to a different cluster.
ImportantAfter you switch the cluster, task execution may vary after submission. Proceed with caution.
Production Environment Engine Version
Defaults to the engine version selected when you created the task. You can switch to a different version.
Development Environment Cluster and Engine Version
You can use the default system configuration or specify a custom configuration.
-
System default configuration: This is the default option. It uses the same cluster and engine version as the production environment.
-
Custom configuration: You can select the cluster and engine version for task execution in the development environment.
NoteIf your project is in Basic mode, you do not need to configure this item.
Parallelism
The default value is 4. You can enter any integer greater than 0. A value of -1 indicates automatic inference.
Number of Task Managers
The default value is 2. You can enter any integer greater than 0.
Task Manager CPUs
The default value is 1. You can enter any number greater than 0, such as 1 or 10.5.
Task Manager Memory
The default value is 8 Gi. Use Gi or Mi as the unit. You can enter a number in bytes, or a number with a memory unit, such as 1024000, 1024 Mi, or 1.5 Gi.
Job Manager CPUs
CPU resources allocated to the task in the Flink cluster control unit. The default value is 1. You can enter any number greater than 0, such as 1 or 10.5.
Job Manager Memory
Memory allocated to the task in the Flink cluster control unit. The default value is 4 Gi. Use Gi or Mi as the unit. You can enter a number in bytes, or a number with a memory unit, such as 1024000, 1024 Mi, or 1.5 Gi.
Resource Configuration Type
-
System-recommended configuration
Intelligent mode (BETA) for fine-grained resource configuration. This mode extends expert mode by enabling the AutoPilot automatic tuning feature.
With AutoPilot enabled, you do not need to manually configure resources. AutoPilot automatically generates and adjusts resource configurations based on job status, optimizing utilization without affecting job health. For more information, see Configure Autopilot.
-
Custom configuration
Expert mode (BETA) for fine-grained resource configuration in fully managed Flink. Provides granular control over job resources to meet throughput requirements.
The system runs the job in Native K8s mode based on your configured resource requirements, and determines the specifications and number of Task Managers (TMs) from the Slot specifications and job concurrency. For more information, see Configure job deployment information.
To customize Flink SQL task resources in Dataphin, see Customize resources for Ververica Flink real-time tasks.
NoteFlink DataStream tasks do not support the Resource Configuration Type configuration.
Resource Configuration Records
When the Resource Configuration information changes, you can click Save Current Configuration as New Record. For a saved resource configuration record, you can View Details, Start, or Delete it.
NoteFlink DataStream tasks do not support Resource Configuration Records.
-
-
Click OK to save the configuration.