Dataphin:Session clusters

Background

Manage cluster resources for Apache Flink jobs running in session mode under the Flink on YARN deployment mode. You can create, view, and manage multiple session cluster instances and monitor their resource configuration and status.

• Resource isolation: Use multiple session clusters to isolate resources for different services or projects.

• Centralized management: Manage all session clusters from a unified interface to improve operational efficiency.

• Status monitoring: View the real-time status and resource usage of your clusters.

• Simplified operations: Use quick actions to create, start, and stop clusters.

Limitations

Session clusters are available only when the real-time compute engine is Apache Flink (open-source Flink) and the deployment mode is Flink on YARN.

Permissions

• A super administrator, system administrator, or user with a custom global role that includes the Project - Cluster Management - Manage permission can create and manage session clusters.

• An owner can manage their assigned cluster.

Create a session cluster

1. In the top navigation bar of the Dataphin homepage, click Plan.

2. In the left-side navigation pane, choose Project > Session Cluster.

3. On the Session Cluster page, click + New Session Cluster.

4. In the New Session Cluster dialog box, configure the following parameters and click OK.

   • Basic information

     Parameter	Description
     Cluster name	Enter a name for the cluster. The name can contain only letters, digits, and underscores (_), and must be no more than 64 characters long.
     Owner	Select a tenant member as the owner. The owner can edit the cluster, view its version history, and delete it.
     Cluster	Select a cluster. Only Cloudera Data Platform 7.x and Aliyun EMR 5.x clusters are supported.
     Engine version	Currently, only Flink 1.20 is supported.
     Queue	Enter the name of the queue.

   • Kerberos authentication configuration

     Parameter	Description
     Flink Kerberos	Enables Kerberos authentication for starting the session cluster. If the selected cluster does not have Kerberos enabled, this option is disabled by default and cannot be enabled. If the selected cluster has Kerberos enabled, this option is enabled by default but can be disabled.
     Keytab file	If you enable Kerberos authentication, you must upload a keytab file.
     Principal	If you enable Kerberos authentication, you must enter the principal. For example, XXXX/hadoopclient@xxx.xxx.
     Username	If you disable Kerberos authentication, you must enter a username. Ensure the user has all permissions required for jobs to run correctly.

   • Resource configuration

     Parameter	Description
     JobManager	Set the CPU and memory for the JobManager. Values must be positive integers. CPU: Minimum is 1 core, default is 1 core. Memory: Minimum is 1 GB, default is 4 GB.
     TaskManager	Set the CPU and memory for the TaskManager. Values must be positive integers. CPU: Minimum is 1 core, default is 2 cores. Memory: Minimum is 1 GB, default is 8 GB.
     Number of slots	Set the number of slots. The value must be a positive integer. Minimum: 1. Default: 8.

   • Runtime parameters

     Enter runtime parameters for the session cluster. These serve as the default configuration when you debug Flink jobs on this cluster.

     Important

     If a runtime parameter conflicts with a job configuration, the job configuration takes precedence.

Manage session clusters

1. In the top navigation bar of the Dataphin homepage, click Plan.

2. In the left-side navigation pane, choose Project > Session Cluster.

3. On the Session Cluster page, view the list of session clusters. The list displays the Cluster Name, Owner, CPU, Memory, Status, and Last Updated By/Time for each cluster.

   Status: The cluster status. Possible values: Starting, Running, Stopping, Stopped, and Failed. If a cluster's status is Failed, you can click the icon to view the error code and message in the Error Message dialog box. You can also copy the error message.

4. In the Actions column for a target cluster, the following actions are available.

   Actions	Description
   View	Click the icon in the Actions column for the target cluster to view the details of its current version.
   Start	Starts a cluster that is Stopped.
   Stop	Stops a cluster that is Running. Important Stopping a session cluster automatically stops all jobs running on it.
   Edit	Click the icon in the Actions column for a cluster in the Stopped status to modify its configuration in the Edit Session Cluster dialog box. After making your changes, click Save. In the dialog box that appears, enter a change description and click OK.
   Go to Flink UI	Opens the Flink UI of a Running cluster to view its details.
   Start/stop logs	View the start and stop logs for the session cluster from the last 7 days. The logs include the operation type, operation information, and log details.
   Delete	Note A session cluster can be deleted only if it is not associated with any compute source. A deleted cluster cannot be recovered. Click the icon in the Actions column for the target cluster and select Delete. In the confirmation dialog box, click OK.