Configure open-source Flink offline mode resources - Dataphin

This topic describes how to configure resources for unified stream and batch tasks that run in offline mode on open-source Flink. You can configure resource queues and engine versions for production and development environments. You can also configure parameters such as the degree of parallelism, Task Manager Slots, Job Manager Memory, and Task Manager Memory.

Procedure

On the Dataphin home page, select Development > Data Development from the top menu bar.
On the Development page, select a project from the top menu bar. If your project is in Dev-Prod mode, also select an environment.
In the navigation pane on the left, navigate to Data Processing > Script Task.
In the script task list, click the target script task to open its tab.
In the right-side sidebar, click Configure. In the configuration panel, click the Offline Mode tab.

In the Resource Configuration section of the configuration panel, configure the Resource Configuration parameters for offline mode.

Flink on Yarn

Parameter	Description
Production Environment Resource Queue	Defaults to the cluster that you selected when you created the Flink SQL task. You can switch the engine here. Important Task execution may differ after you switch the engine and submit the task. Proceed with caution.
Production Environment Engine Version	Defaults to the engine version that you selected when you created the Flink SQL task. You can switch the engine version here.
Development Environment Resource Queue And Engine Version	Defaults to the configuration that you selected when you created the Flink SQL task. You can reconfigure it here. You can select System Default Configuration or Custom Configuration. System Default Configuration: Uses the same cluster and engine version as the production environment. Custom Configuration: You can select the cluster and engine version for the development environment task. This configuration is not required if your project is in Basic mode.
Parallelism	The default value is 1. You can enter any integer greater than 0. A value of -1 indicates automatic inference.
Task Manager Slots	The default value is empty. You can enter any integer greater than 0.
Job Manager Memory	The amount of memory that the task's control unit uses in the Flink cluster. The default value is 1 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.
Task Manager Memory	The default value is 1 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.
Job Manager CPUs	The CPU computing resources assigned to the Job Manager process. The default value is 1. You can enter any integer greater than 0.
JobManager CPUs	The CPU computing resources assigned to the Task Manager process. The default value is 1. You can enter any integer greater than 0.

Flink on Kubernetes

Parameter	Description
Production Environment Resource Queue	Defaults to the cluster that you selected when you created the Flink SQL task. You can switch the engine here. Important Task execution may differ after you switch the engine and submit the task. Proceed with caution.
Production Environment Engine Version	Defaults to the engine version that you selected when you created the Flink SQL task. You can switch the engine version here.
Development Environment Resource Queue And Engine Version	Defaults to the configuration that you selected when you created the Flink SQL task. You can reconfigure it here. You can select System Default Configuration or Custom Configuration. System Default Configuration: Uses the same cluster and engine version as the production environment. Custom Configuration: You can select the cluster and engine version for the development environment task. This configuration is not required if your project is in Basic mode.
Parallelism	The default value is 1. You can enter any integer greater than 0. A value of -1 indicates automatic inference.
Number Of Task Managers	The default value is 1. You can enter any integer greater than 0.
Job Manager CPUs	The amount of CPU resources that the task's control unit uses in the Flink cluster. The default value is 1 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.
Task Manager Memory	The default value is 1 Gi. Use Gi or Mi as the unit. You can enter a number in bytes or a number with a memory unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.

Click OK to complete the resource configuration for the offline task.