All Products
Search
Document Center

Realtime Compute for Apache Flink:Create a deployment

Last Updated:Jan 08, 2024

After you develop a draft, you must publish the draft as a deployment. Then, you can start and run the deployment. This topic describes how to create an SQL deployment, a JAR deployment, and a Python deployment.

Prerequisites

  • A Python package is developed before you create a Python deployment. For more information about how to develop a Python package and the related limits, see Develop a Python API draft.

  • A JAR package is developed before you create a JAR deployment. For more information about how to develop a JAR package, the related limits, and precautions, see Develop a JAR draft.

Limits

Only Realtime Compute for Apache Flink that uses Ververica Runtime (VVR) 4.0.0 or later allows you to create a Python deployment.

Upload resources

Before you create a deployment, perform the following steps to upload a JAR package, a Python deployment file, or a Python dependency file to the console of fully managed Flink.

Note

A maximum of 200 packages can be uploaded. The size of each package cannot exceed 200 MB. If the size of the JAR package exceeds 200 MB, we recommend that you upload the package in the Object Storage Service (OSS) console. For more information, see Reference.

  1. Log on to the Realtime Compute for Apache Flink console.

  2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.

  3. In the left-side navigation pane, click Artifacts.

  4. On the Artifacts page, click Upload Artifact. Select the JAR file, Python deployment file, or Python dependency that you want to upload.

Note

If your deployment is a Python API deployment, upload the official JAR file of PyFlink. To download the official JAR file of the required version, click PyFlink V1.11 or PyFlink V1.12.

Procedure

  1. Log on to the Realtime Compute for Apache Flink console.

  2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column. You can perform the following steps based on the type of deployment that you want to create.

    Create an SQL deployment

    1. On the SQL Editor page, develop an SQL draft. For more information, see Develop an SQL draft.

    2. After the SQL draft is developed, click Deploy in the upper-right corner of the SQL Editor page.

    3. In the Create Deployment dialog box, configure the parameters. The following table describes the parameters.

      Parameter

      Description

      Comment

      Optional. You can enter the description for the deployment.

      Label

      After you specify labels for a deployment, you can search for the deployment by label key and label value on the Deployments page. You can specify a maximum of three labels for a deployment.

      Deployment Target

      Select the desired queue or session cluster from the drop-down list. We recommend that you do not use session clusters in the production environment. For more information, see Manage queues and Step 1: Create a session cluster.

      Note

      Metrics of deployments that are deployed in session clusters cannot be displayed. Session clusters do not support the monitoring and alerting feature and the Autopilot feature. Session clusters are suitable for development and test environments. We recommend that you do not use session clusters in the production environment. For more information, see Debug a deployment.

      Skip validation on draft before deploying

      If you select this option, you can skip a syntax check before you deploy a draft as a deployment.

    4. In the Deploy draft dialog box, click Confirm.

    Note

    After the deployment operation is complete, start the deployment on the Deployments page. For more information about how to start a deployment, see Start a deployment.

    Create a JAR deployment

    1. In the left-side navigation pane, click Deployments. On the Deployments page, click Create Deployment.

    2. In the Create Deployment dialog box, configure the parameters. The following table describes the parameters.

      Parameter

      Description

      Deployment Type

      Select JAR.

      Deployment Mode

      Select Stream Mode or Batch Mode.

      Deployment Name

      Enter the name of the JAR deployment that you want to create.

      Engine Version

      For more information about engine versions, see Engine version and Lifecycle policies. We recommend that you use a recommended version or a stable version.

      • Recommend: the latest minor version of the current latest major version.

      • Stable: the latest minor version of a major version that is still in the service period of the product. Defects in previous versions are fixed in such a version.

      • Normal: other minor versions that are still in the service period of the product.

      • Deprecated: the version that exceeds the service period of the product.

      Note

      In VVR 3.0.3 and later, Ververica Platform (VVP) allows you to run SQL deployments that use different engine versions at the same time. The version of the Flink engine that uses VVR 3.0.3 is Flink 1.12. If the engine version of your deployment is Flink 1.12 or earlier, you can perform the following operations to update the engine version based on the engine version that your deployment uses:

      • Flink 1.12: Stop and then restart your deployment. Then, the system automatically updates the engine version of your deployment to vvr-3.0.3-flink-1.12.

      • Flink 1.11 or Flink 1.10: Manually update the engine version of your deployment to vvr-3.0.3-flink-1.12 or vvr-4.0.8-flink-1.13, and then restart the deployment. Otherwise, a timeout error occurs when you start the deployment.

      JAR URI

      Select a file or manually upload a new file. You can drag the file that you want to upload to this field or click the 上传 icon on the right to select the file that you want to upload.

      Note

      If your deployment is a Python API deployment, upload the official JAR file of PyFlink. To download the official JAR file of the required version, click PyFlink V1.11 or PyFlink V1.12.

      Entry Point Class

      The entry point class of the program. If you do not specify a main class for the JAR file, enter a standard path in the Entry Point Class field.

      Note

      If your deployment is a Python API deployment, set Entry Point Class to org.apache.flink.client.python.PythonDriver.

      Entry Point Main Arguments

      You can pass parameters and call them in the main method.

      Note
      • The parameter information cannot be greater than 1,024 characters in length. We recommend that you do not use complex parameters. The parameters that include line breaks, spaces, or other special characters are considered complex parameters. If you want to pass complex parameters, use a dependency file.

      • If your deployment is a Python API deployment, upload the Python deployment file first. By default, after you upload the Python deployment file, the file is uploaded to the /flink/usrlib/ directory of the node that runs the deployment.

        If the Python deployment file is named word_count.py, set Entry Point Main Arguments to -py /flink/usrlib/word_count.py.

        Enter the full path of the Python deployment file. You cannot omit or change /flink/usrlib/.

      Additional Dependencies

      • (Recommended) Select the dependency file that you uploaded.

        You must upload a dependency file in advance. To upload a dependency file, click Upload Artifacts in the upper-left corner of the Artifacts page or click the 更新JAR icon on the right side of Additional Dependencies in the Create Deployment dialog box. The uploaded dependency file is saved in the oss://ossBucketName/artifacts/namespaces/namespaceName/<file> directory.

      • Enter the OSS bucket where the required dependency file is stored.

        You must upload the dependency file to the OSS bucket that corresponds to the current instance in advance. The OSS bucket to which the dependency file is uploaded must be the OSS bucket that you selected to activate fully managed Flink.

      • Enter the URL of the required dependency file. Only URLs that end with file names are supported, such as http://xxxxxx/<file>.

        You must upload the dependency file to the publicly accessible HTTP service in advance.

      Note
      • The dependency file that you upload by using one of the preceding methods can also be downloaded to the destination machine. When the deployment is running, the dependency file is loaded to the /flink/usrlib directory of the pod in which the JobManager and TaskManager reside.

      • If you select Deploy to Session Cluster (not recommended) when you create the deployment, you cannot specify the path of the dependency file for the deployment.

      Deployment Target

      Select the desired queue or session cluster from the drop-down list. We recommend that you do not use session clusters in the production environment. For more information, see Manage queues and Step 1: Create a session cluster.

      Note

      Metrics of deployments that are deployed in session clusters cannot be displayed. Session clusters do not support the monitoring and alerting feature and the Autopilot feature. Session clusters are suitable for development and test environments. We recommend that you do not use session clusters in the production environment. For more information, see Debug a deployment.

      Description

      Optional. You can enter the description for the deployment.

      Label

      After you specify labels for a deployment, you can search for the deployment by label key and label value on the Deployments page. You can specify a maximum of three labels for a deployment.

      More Setting

      If you turn on the switch, you must configure the following parameters:

      • Kerberos Name: Select a Hive cluster that supports Kerberos authentication from the drop-down list. For more information about how to create a Hive cluster that supports Kerberos authentication, see Register a Hive cluster that supports Kerberos authentication.

      • principal: a Kerberos principal, which can be a user or a service. A Kerberos principal is used to uniquely identify an identity in the Kerberos encryption system.

    3. In the Create Deployment dialog box, click Deploy.

    Note

    After the deployment operation is complete, start the deployment on the Deployments page. For more information about how to start a deployment, see Start a deployment.

    Create a Python deployment

    1. In the left-side navigation pane, click Deployments. On the Deployments page, click Create Deployment.

    2. In the Create Deployment dialog box, configure the parameters. The following table describes the parameters.

      Parameter

      Description

      Deployment Type

      Select PYTHON.

      Deployment Mode

      Select Stream Mode or Batch Mode.

      Deployment Name

      Enter the name of the deployment that you want to create.

      Engine Version

      For more information about engine versions, see Engine version and Lifecycle policies. We recommend that you use a recommended version or a stable version.

      • Recommend: the latest minor version of the current latest major version.

      • Stable: the latest minor version of a major version that is still in the service period of the product. Defects in previous versions are fixed in such a version.

      • Normal: other minor versions that are still in the service period of the product.

      • Deprecated: the version that exceeds the service period of the product.

      Note

      In VVR 3.0.3 and later, VVP allows you to run Python deployments that use different engine versions at the same time. The version of the Flink engine that uses VVR 3.0.3 is Flink 1.12. If the engine version of your deployment is Flink 1.12 or earlier, you can perform the following operations to update the engine version based on the engine version that your deployment uses:

      • Flink 1.12: Stop and then restart your deployment. Then, the system automatically updates the engine version of your deployment to vvr-3.0.3-flink-1.12.

      • Flink 1.11 or Flink 1.10: Manually update the engine version of your deployment to vvr-3.0.3-flink-1.12 or vvr-4.0.8-flink-1.13, and then restart the deployment. Otherwise, a timeout error occurs when you start the deployment.

      Python Uri

      The Uniform Resource Identifier (URI) used to access the Python deployment file that you want to upload. Python deployment files can be .py files or .zip files.

      Entry Module

      The entry point class of the program. If the Python deployment file that you select is a .py file, you do not need to configure this parameter. If the Python deployment file that you select is a .zip file, you must configure this parameter. For example, you can enter example.word_count in the Entry Module field.

      Entry Point Main Arguments

      The parameters of the deployment.

      Python Libraries

      A third-party Python package. The third-party Python package that you uploaded is added to PYTHONPATH of the Python worker process so that the package can be directly accessed in Python user-defined functions (UDFs). For more information about how to use third-party Python packages, see Use a third-party Python package.

      Python Archives

      Archive files. Only ZIP files such as .zip, .jar, .whl, and .egg are supported.

      Archive files are decompressed to the working directory of the Python worker process. For example, if the name of the compressed file where the archive files are located is mydata.zip, the following code can be written in Python UDFs to access the mydata.zip archive file.

      def map():  
          with open("mydata.zip/mydata/data.txt") as f: 
          ...

      For more information about Python Archives, see Use a custom Python virtual environment and Use data files.

      Additional Dependencies

      You can upload a file for the deployment, such as a Python deployment file and a data file that the deployment requires. For more information about Python dependencies, see Manage Python dependencies. By default, the dependency file that you upload is downloaded to the /flink/usrlib/ directory of the node on which the deployment runs.

      Note

      Session clusters do not support the configuration of the Additional Dependencies parameter. Only per-job clusters support the configuration of the Additional Dependencies parameter.

      Deployment Target

      Select the desired queue or session cluster from the drop-down list. We recommend that you do not use session clusters in the production environment. For more information, see Manage queues and Step 1: Create a session cluster.

      Note

      Metrics of deployments that are deployed in session clusters cannot be displayed. Session clusters do not support the monitoring and alerting feature and the Autopilot feature. Session clusters are suitable for development and test environments. We recommend that you do not use session clusters in the production environment. For more information, see Debug a deployment.

      Description

      Optional. You can enter the description for the deployment.

      Label

      After you specify labels for a deployment, you can search for the deployment by label key and label value on the Deployments page. You can specify a maximum of three labels for a deployment.

      More Setting

      If you turn on the switch, you must configure the following parameters:

      • Kerberos Name: Select a Hive cluster that supports Kerberos authentication from the drop-down list. For more information about how to create a Hive cluster that supports Kerberos authentication, see Register a Hive cluster that supports Kerberos authentication.

      • principal: a Kerberos principal, which can be a user or a service. A Kerberos principal is used to uniquely identify an identity in the Kerberos encryption system.

    3. In the Create Deployment dialog box, click Deploy.

    Note

    After the deployment operation is complete, start the deployment on the Deployments page. For more information about how to start a deployment, see Start a deployment.