Configure resources for open-source Flink in real-time mode - Dataphin

This topic describes how to configure resources for open-source Flink stream-batch integrated tasks in real-time mode. You can configure settings such as resource queues, engine versions, degree of parallelism, Task Manager Slots, Job Manager Memory, and Task Manager Memory.

Procedure

In the top menu bar of the Dataphin home page, choose Developer > Data Development.
In the top menu bar, select a project. If you are in Dev-Prod mode, you must also select an environment.
In the left navigation pane, choose Data Processing > Script Task.
In the list of computing tasks, select the target open-source Flink task and click Configuration in the right sidebar.

In the configuration panel, click the Real-time Mode tab, and configure the settings in the Real-time Mode tab's Resource Configuration section.

Flink on yarn

Parameter	Description
Production Environment Resource Queue	The resource queue selected when you created the Flink SQL task is used by default. You can switch the DPI engine here. Important After you switch the engine, the execution of submitted tasks may vary. Proceed with caution.
Production Environment Engine Version	The engine version selected when you created the Flink SQL task is used by default. You can switch the engine version here. Supported versions are 1.13.1, 1.14.2, 1.15.3, and 1.20.1.
Development Environment Resource Queue And Engine Version	The configuration selected when you created the Flink SQL task is used by default. You can reconfigure it here. You can select System Default Configuration or Custom Configuration. System Default Configuration: Uses the same cluster and engine version as the production environment. Custom Configuration: You can select the Development Environment Resource Queue and Development Environment Engine Version. The available versions are 1.13.1, 1.14.2, 1.15.3, and 1.20.1. If your project is in Basic mode, you do not need to configure this item.
Parallelism	The default value is 1. You can enter any integer greater than 0. A value of -1 indicates automatic inference.
Task Manager Slots (Optional)	This parameter is empty by default. You can enter any integer greater than 0.
Job Manager Memory	The amount of memory that the task occupies in the control unit of the Flink cluster. The default value is 2 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. Examples: 1024000, 1024Mi, and 1.5Gi.
Task Manager Memory	The default value is 2 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. Examples: 1024000, 1024Mi, and 1.5Gi.
Job Manager CPUs	The CPU computing resources assigned to the JobManager process. The default value is 1C. You can enter any integer greater than 0.
Task Manager CPUs	The CPU computing resources assigned to the TaskManager process. The default value is 1C. You can enter any integer greater than 0.

Flink on K8s

Parameter	Description
Production Environment Resource Queue	The resource queue selected when you created the Flink SQL task is used by default. You can switch the DPI engine here. Important After you switch the engine, the execution of submitted tasks may vary. Proceed with caution.
Production Environment Engine Version	The engine version selected when you created the Flink SQL task is used by default. You can switch the engine version here. Supported versions are 1.13.1, 1.14.2, 1.15.3, and 1.20.1.
Development Environment Resource Queue And Engine Version	The configuration selected when you created the Flink SQL task is used by default. You can reconfigure it here. You can select System Default Configuration or Custom Configuration. System Default Configuration: Uses the same cluster and engine version as the production environment. Custom Configuration: You can select the Development Environment Resource Queue and Development Environment Engine Version. The available versions are 1.13.1, 1.14.2, 1.15.3, and 1.20.1. If your project is in Basic mode, you do not need to configure this item.
Parallelism	The default value is 1. You can enter any integer greater than 0. A value of -1 indicates automatic inference.
Task Manager Count	The default value is 1. You can enter any integer greater than 0.
Job Manager Memory	The amount of memory that the task occupies in the control unit of the Flink cluster. The default value is 1 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. Examples: 1024000, 1024Mi, and 1.5Gi.
Task Manager Memory	The default value is 1 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. Examples: 1024000, 1024Mi, and 1.5Gi.
JobManager CPU	The CPU computing resources assigned to the JobManager process. The default value is 1C. You can enter any number greater than 0, such as 1 or 10.5.
TaskManager CPU	The CPU computing resources assigned to the TaskManager process. The default value is 2C. You can enter any number greater than 0, such as 1 or 10.5.

Click OK to complete the resource configuration.