This topic outlines the configuration of Ververica Flink resources for stream-batch integrated tasks in real-time mode, including settings for production and development clusters, engine versions, parallelism, and CPU and memory for Task Managers and Job Managers.
Procedure
Navigate to the Development section from the top menu bar on the Dataphin home page. By default, you will be directed to the data development page.
To access the stream-batch integrated task real-time mode configuration panel, follow the steps below:
Select the project (Dev-Prod mode requires selecting the environment) -> single click Script Task -> choose and single click the computing task -> single click Configure -> single click Real-time Mode.
In the Resource Configuration section of the real-time mode configuration panel, set the Resource Configuration details.
Parameter
Description
Production Environment Cluster
The cluster initially selected during task creation is used by default. You have the option to switch to a different cluster here.
ImportantSwitching clusters may result in variations in task execution upon submission. Exercise caution when making changes.
Production Environment Engine Version
The engine version selected during task creation is used by default. You can update the engine version here.
Development Environment Cluster And Engine Version
You can select between the system default configuration, which uses the same cluster and engine version as the production environment, or a custom configuration for the development environment task.
System default configuration: Automatically selected, matching the production environment's cluster and engine version.
Custom configuration: Allows you to specify the cluster and engine version for the development environment task execution.
NoteIf your project space is in Basic mode, this configuration is not required.
Parallelism
The default parallelism is set to 4, but you can specify any integer greater than 0. A value of -1 indicates automatic inference.
Task Manager Quantity
The default number of Task Managers is 2, but you can enter any integer greater than 0.
Task Manager CPUs
The default CPU allocation for each Task Manager is 1, but you can specify any positive number, such as 1 or 10.5.
Task Manager Memory
The default memory allocation for Task Managers is 8Gi. It is recommended to use Gi/Mi units; you can enter numbers in bytes or include memory units such as 1024000, 1024Mi, or 1.5Gi.
Job Manager Cpus
The default CPU allocation for the Job Manager is 1, but you can specify any positive number, such as 1 or 10.5.
Task Manager Memory
The default memory allocation for the Job Manager is 4Gi. Use Gi/Mi units for specification; you can enter numbers in bytes or include memory units such as 1024000, 1024Mi, or 1.5Gi.
Resource Configuration Type
System Recommended Configuration
This intelligent mode (BETA) for fine-grained resource configuration of real-time computing Flink is an advanced version of expert mode. It includes the AutoPilot automatic tuning function alongside the expert mode's resource configuration.
In intelligent mode, AutoPilot automatically generates resource configurations for the job and optimizes them based on the job's running status, ensuring optimal resource usage and job health. For more information, see AutoPilot configuration.
Custom Configuration
Expert mode (BETA) allows for detailed resource configuration of real-time computing Flink tasks, providing fine-grained control to meet specific throughput requirements.
Based on the configured resources, the system will automatically deploy the job in Native K8s mode, determining the specifications and quantity of Task Managers (TMs) according to the Slot specifications and job concurrency. For details on fine-grained resource configuration, see Configure job deployment information.
To customize Flink SQL task resources in Dataphin, refer to Ververica Flink real-time task resource customization.
NoteFLINK_DATASTREAM tasks do not support the Resource Configuration Type setting.
Resource Configuration Record
Upon changes to the Resource Configuration, you may Save the Current Configuration as a New Record with a single click. For the saved resource configuration record, options to view details, start, or delete are available.
NoteFLINK_DATASTREAM tasks do not support the Resource Configuration Record feature.
To finalize the real-time mode resource configuration for the task, single click Confirm.