Dataphin:Session clusters

Background

The session cluster feature allows you to manage cluster resources for Apache Flink jobs that run in session mode with a Flink on YARN deployment mode. You can create, view, and manage multiple session cluster instances, and monitor their resource configuration and status.

• Resource isolation: Use multiple session clusters to isolate resources for different services or projects.

• Centralized management: Manage all session clusters from a unified interface to improve operational efficiency.

• Status monitoring: View the real-time status and resource usage of your clusters.

• Simplified operations: Use quick actions to create, start, and stop clusters.

Limitations

The session cluster feature is available only when the real-time compute engine is Apache Flink (open-source Flink) and the deployment mode is Flink on YARN.

Permissions

• A super administrator, system administrator, or user with a custom global role that includes the Project - Cluster Management - Manage permission can create and manage session clusters.

• An owner can manage their assigned cluster.

Create a session cluster

1. In the top navigation bar of the Dataphin homepage, click Plan.

2. In the left-side navigation pane, choose Project > Session Cluster.

3. On the Session Cluster page, click + New Session Cluster.

4. In the New Session Cluster dialog box, configure the following parameters and click OK.

   • Basic information

     Parameter	Description
     Cluster name	Enter a name for the cluster. The name can contain only letters, digits, and underscores (_), and must be no more than 64 characters long.
     Owner	Select a tenant member as the owner. The owner has permissions to edit, view the cluster's version history, and delete it.
     Cluster	Select a cluster. Only Cloudera Data Platform 7.x and Aliyun EMR 5.x clusters are supported.
     Engine version	Currently, only Flink 1.20 is supported.
     Queue	Enter the name of the queue.

   • Kerberos authentication configuration

     Parameter	Description
     Flink Kerberos	Enables Kerberos authentication for starting the session cluster. If Kerberos authentication is not enabled on the selected cluster, this option is disabled by default and cannot be enabled. If Kerberos authentication is enabled on the selected cluster, this option is enabled by default but can be disabled.
     Keytab file	If you enable Kerberos authentication, you must upload a keytab file.
     Principal	If you enable Kerberos authentication, you must enter the principal. For example, XXXX/hadoopclient@xxx.xxx.
     Username	If you disable Kerberos authentication, you must enter a username. To ensure that jobs run correctly, confirm that the user has all required permissions.

   • Resource configuration

     Parameter	Description
     JobManager	Specify the CPU and memory for the session cluster's JobManager. Values must be positive integers. CPU: Minimum is 1 core, default is 1 core. Memory: Minimum is 1 GB, default is 4 GB.
     TaskManager	Specify the CPU and memory for the session cluster's TaskManager. Values must be positive integers. CPU: Minimum is 1 core, default is 2 cores. Memory: Minimum is 1 GB, default is 8 GB.
     Number of slots	Specify the number of slots. The value must be a positive integer. The minimum is 1, and the default is 8.

   • Runtime parameters

     You can enter runtime parameters for the session cluster. These parameters are used as the default configuration when you debug Flink jobs using this session cluster.

     Important

     If a runtime parameter conflicts with a job configuration, the job configuration takes precedence.

Manage session clusters

1. In the top navigation bar of the Dataphin homepage, click Plan.

2. In the left-side navigation pane, choose Project > Session Cluster.

3. On the Session Cluster page, view the list of session clusters. The list displays the Cluster Name, Owner, CPU, Memory, Status, and Last Updated By/Time for each cluster.

   Status: Indicates the status of the session cluster. Possible statuses are Starting, Running, Stopping, Stopped, and Failed. If a cluster's status is Failed, you can click the icon to view the error code and message in the Error Message dialog box. You can also copy the error message.

4. In the Actions column for a target cluster, the following actions are available.

   Actions	Description
   View	Click the icon in the Actions column for the target cluster to view the details of its current version.
   Start	Starts a cluster that is Stopped.
   Stop	Stops a cluster that is Running. Important Stopping a session cluster automatically stops all jobs running on it.
   Edit	Click the icon in the Actions column for a cluster in the Stopped status to modify its configuration in the Edit Session Cluster dialog box. After making your changes, click Save. In the dialog box that appears, enter a change description and click OK.
   Go to Flink UI	Navigates to the Flink UI of a Running cluster to view its details.
   Start/stop logs	View the start/stop logs for the session cluster from the last 7 days. The logs include the operation type, operation information, and log details.
   Delete	Note A session cluster can be deleted only if it is not associated with any compute source. A deleted cluster cannot be recovered. Click the icon in the Actions column for the target cluster and select Delete. In the confirmation dialog box, click OK.