Configure resource queues, engine versions, and runtime parameters such as parallelism, Task Manager Slots, Job Manager Memory, and Task Manager Memory for unified stream and batch tasks running in open-source Flink offline mode.
Procedure
-
On the Dataphin home page, select Development > Data Development from the top menu bar.
-
On the Development page, select a project from the top menu bar. If your project is in Dev-Prod mode, also select an environment.
-
In the navigation pane on the left, navigate to Data Processing > Script Task.
-
In the script task list, click the target script task to open its tab.
-
In the right-side sidebar, click Configure. In the configuration panel, click the Offline Mode tab.
-
In the Resource Configuration section of the configuration panel, configure the Resource Configuration parameters for offline mode.
Flink on Yarn
Parameter
Description
Production Environment Resource Queue
Defaults to the cluster that you selected when you created the Flink SQL task. You can switch the engine here.
ImportantTask execution may differ after you switch the engine and submit the task. Proceed with caution.
Production Environment Engine Version
Defaults to the engine version that you selected when you created the Flink SQL task. You can switch the engine version here.
Development Environment Resource Queue And Engine Version
Defaults to the configuration that you selected when you created the Flink SQL task. Select System Default Configuration or Custom Configuration.
-
System Default Configuration: Uses the same cluster and engine version as the production environment.
-
Custom Configuration: You can select the cluster and engine version for the development environment task. This configuration is not required if your project is in Basic mode.
Parallelism
Default: 1. Enter any integer greater than 0. A value of -1 indicates automatic inference.
Task Manager Slots
Default: empty. Enter any integer greater than 0.
Job Manager Memory
Memory allocated to the Job Manager in the Flink cluster. Default: 1 Gi. Specify a value in bytes or with a unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.
Task Manager Memory
Default: 1 Gi. Specify a value in bytes or with a unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.
Job Manager CPUs
CPU cores assigned to the Job Manager process. Default: 1. Enter any integer greater than 0.
JobManager CPUs
CPU cores assigned to the Task Manager process. Default: 1. Enter any integer greater than 0.
Flink on Kubernetes
Parameter
Description
Production Environment Resource Queue
Defaults to the cluster that you selected when you created the Flink SQL task. You can switch the engine here.
ImportantTask execution may differ after you switch the engine and submit the task. Proceed with caution.
Production Environment Engine Version
Defaults to the engine version that you selected when you created the Flink SQL task. You can switch the engine version here.
Development Environment Resource Queue And Engine Version
Defaults to the configuration that you selected when you created the Flink SQL task. Select System Default Configuration or Custom Configuration.
-
System Default Configuration: Uses the same cluster and engine version as the production environment.
-
Custom Configuration: You can select the cluster and engine version for the development environment task. This configuration is not required if your project is in Basic mode.
Parallelism
Default: 1. Enter any integer greater than 0. A value of -1 indicates automatic inference.
Number Of Task Managers
Default: 1. Enter any integer greater than 0.
Job Manager CPUs
CPU resources assigned to the Job Manager in the Flink cluster. Default: 1 Gi. Specify a value in bytes or with a unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.
Task Manager Memory
Default: 1 Gi. Specify a value in bytes or with a unit such as Gi or Mi. For example, 1024000, 1024Mi, or 1.5Gi.
-
-
Click OK to complete the resource configuration for the offline task.