After you develop a draft, you must publish the draft as a deployment. This way, development is isolated from production. The deployments that are running are not affected. A draft is officially published only after the deployment for the draft is started or restarted. This topic describes how to create an SQL deployment, a JAR deployment, and a Python deployment.
Prerequisites
A draft is developed.
An SQL draft is developed before you create an SQL deployment. For more information, see Develop an SQL draft.
A Python package is developed before you create a Python deployment. For more information, see Develop a Python API draft.
A JAR package is developed before you create a JAR deployment. For more information, see Develop a JAR draft.
Limits
Only Realtime Compute for Apache Flink that uses Ververica Runtime (VVR) 4.0.0 or later allows you to create a Python deployment.
Upload resources
Before you create a deployment, upload a JAR package, a Python deployment file, or a Python dependency file to the development console of Realtime Compute for Apache Flink based on your business requirements.
Log on to the Realtime Compute for Apache Flink console.
Find the workspace that you want to manage and click Console in the Actions column.
In the left-side navigation pane, click Artifacts.
On the Artifacts page, click Upload Artifact. Select the JAR file, Python deployment file, or Python dependency file that you want to upload.
If your deployment is a Python API deployment, upload the official JAR file of PyFlink. To download the official JAR file of the required version, click PyFlink V1.11 or PyFlink V1.12.
Procedure
Log on to the Realtime Compute for Apache Flink console.
Find the workspace that you want to manage and click Console in the Actions column. You can perform the following steps based on the type of deployment that you want to create.
Create an SQL deployment
On the Development > ETL page, develop an SQL draft. For more information, see Develop an SQL draft.
After the SQL draft is developed, click Deploy in the upper-right corner of the SQL Editor page.
In the Deploy draft dialog box, configure the parameters. The following table describes the parameters.
Parameter
Description
Comment
Optional. Enter the description for the deployment.
Label
After you specify labels for a deployment, you can search for the deployment by label key and label value on the O&M > Deployments page. You can specify a maximum of three labels for a deployment.
Deployment Target
Select the desired queue or session cluster from the drop-down list. We recommend that you do not use session clusters in the production environment. For more information, see Manage queues and Step 1: Create a session cluster.
NoteMetrics of deployments that are deployed in session clusters cannot be displayed. Session clusters do not support the monitoring and alerting feature and the Autopilot feature. Session clusters are suitable for development and test environments. We recommend that you do not use session clusters in the production environment. For more information, see Debug a deployment.
Skip validation on draft before deploying
If you select this option, you can skip a syntax check before you deploy a draft as a deployment.
In the Deploy draft dialog box, click Confirm.
Create a JAR deployment
In the left-side navigation pane, click O&M > Deployments. On the Deployments page, click Create Deployment > JAR Deployment.
In the Create Deployment dialog box, configure the parameters. The following table describes the parameters.
Parameter
Description
Deployment Mode
Select Stream Mode or Batch Mode.
Deployment Name
Enter the name of the JAR deployment that you want to create.
Engine Version
For more information about engine versions, see Engine version and Lifecycle policies. We recommend that you use a recommended version or a stable version.
Recommended: the latest minor version of the latest major version.
Stable: the latest minor version of a major version that is still in the service period of the product. Defects in previous versions are fixed in such a version.
Normal: other minor versions that are still in the service period of the product.
Deprecated: the versions that exceed the service period of the product.
NoteIn VVR 3.0.3 and later, Ververica Platform (VVP) allows you to run JAR deployments that use different engine versions at the same time. The version of the Flink engine that uses VVR 3.0.3 is Flink 1.12. If the engine version of your deployment is Flink 1.12 or earlier, you can perform the following operations to update the engine version based on the engine version that your deployment uses:
Flink 1.12: Stop and then restart your deployment. Then, the system automatically updates the engine version of your deployment to vvr-3.0.3-flink-1.12.
Flink 1.11 or Flink 1.10: Manually update the engine version of your deployment to vvr-3.0.3-flink-1.12 or vvr-4.0.8-flink-1.13, and then restart the deployment. Otherwise, a timeout error occurs when you start the deployment.
JAR URI
Select a file or manually upload a new file. You can drag the file that you want to upload to this field or click the icon on the right to select the file that you want to upload.
NoteIf your deployment is a Python API deployment, upload the official JAR file of PyFlink. To download the official JAR file of the required version, click PyFlink V1.11 or PyFlink V1.12.
Entry Point Class
The entry point class of the program. If you do not specify a main class for the JAR file, enter a standard path in the Entry Point Class field.
NoteIf your deployment is a Python API deployment, set Entry Point Class to org.apache.flink.client.python.PythonDriver.
Entry Point Main Arguments
You can pass parameters and call them in the main method.
NoteThe parameter information cannot be greater than 1,024 characters in length. We recommend that you do not use complex parameters. The parameters that include line breaks, spaces, or other special characters are considered complex parameters. If you want to pass complex parameters, use a dependency file.
If your deployment is a Python API deployment, upload the Python deployment file first. By default, after you upload the Python deployment file, the file is uploaded to the /flink/usrlib/ directory of the node that runs the deployment.
If the Python deployment file is named word_count.py, set Entry Point Main Arguments to
-py /flink/usrlib/word_count.py
.Enter the full path of the Python deployment file. You cannot omit or change /flink/usrlib/.
Additional Dependencies
(Recommended) Select the dependency file that you uploaded.
You must upload a dependency file in advance. To upload a dependency file, click Upload Artifacts in the upper-left corner of the Artifacts page in the development console of Realtime Compute for Apache Flink. Alternatively, click the icon on the right side of Additional Dependencies in the Create Deployment dialog box. The dependency file that you upload is stored in the artifacts directory of the OSS bucket that you select to associate with your Realtime Compute for Apache Flink workspace when you activate the workspace. The file directory is in the format of
oss://<Name of the OSS bucket that is associated with your workspace>/artifacts/namespaces/<Name of the namespace>
.Enter the OSS bucket where the required dependency file is stored.
The OSS bucket to which the dependency file is uploaded must be the OSS bucket that you selected when you activate the current Realtime Compute for Apache Flink workspace.
Enter the URL of the required dependency file.
You must enter the URL of an external storage system that Realtime Compute for Apache Flink can access and is allowed to access. The access control list (ACL) of the external storage system must be public-read or Realtime Compute for Apache Flink must have the permission to access the external storage system. Only URLs that end with file names are supported, such as http://xxxxxx/<file>.
NoteThe dependency file that you upload by using one of the preceding methods can also be downloaded to the destination machine. When the deployment is running, the dependency file is loaded to the /flink/usrlib directory of the pod in which the JobManager and TaskManager reside.
If you select a session cluster for Deployment Target, you cannot configure the dependency file for the deployment.
Deployment Target
Select the desired queue or session cluster from the drop-down list. We recommend that you do not use session clusters in the production environment. For more information, see Manage queues and Step 1: Create a session cluster.
NoteMetrics of deployments that are deployed in session clusters cannot be displayed. Session clusters do not support the monitoring and alerting feature and the Autopilot feature. Session clusters are suitable for development and test environments. We recommend that you do not use session clusters in the production environment. For more information, see Debug a deployment.
Description
Optional. Enter the description for the deployment.
Label
After you specify labels for a deployment, you can search for the deployment by label key and label value on the Deployments page. You can specify a maximum of three labels for a deployment.
More Setting
If you turn on the switch, you must configure the following parameters:
Kerberos Name: Select a Hive cluster that supports Kerberos authentication from the drop-down list. For more information about how to create a Hive cluster that supports Kerberos authentication, see Register a Hive cluster that supports Kerberos authentication.
principal: a Kerberos principal, which can be a user or a service. A Kerberos principal is used to uniquely identify an identity in the Kerberos encryption system.
In the Create Deployment dialog box, click Deploy.
Create a Python deployment
In the left-side navigation pane, click O&M > Deployments. On the Deployments page, click Create Deployment > Python Deployment.
In the Create Deployment dialog box, configure the parameters. The following table describes the parameters.
Parameter
Description
Deployment Mode
Select Stream Mode or Batch Mode.
Deployment Name
Enter the name of the deployment that you want to create.
Engine Version
For more information about engine versions, see Engine version and Lifecycle policies. We recommend that you use a recommended version or a stable version.
Recommended: the latest minor version of the latest major version.
Stable: the latest minor version of a major version that is still in the service period of the product. Defects in previous versions are fixed in such a version.
Normal: other minor versions that are still in the service period of the product.
Deprecated: the versions that exceed the service period of the product.
NoteIn VVR 3.0.3 and later, VVP allows you to run Python deployments that use different engine versions at the same time. The version of the Flink engine that uses VVR 3.0.3 is Flink 1.12. If the engine version of your deployment is Flink 1.12 or earlier, you can perform the following operations to update the engine version based on the engine version that your deployment uses:
Flink 1.12: Stop and then restart your deployment. Then, the system automatically updates the engine version of your deployment to vvr-3.0.3-flink-1.12.
Flink 1.11 or Flink 1.10: Manually update the engine version of your deployment to vvr-3.0.3-flink-1.12 or vvr-4.0.8-flink-1.13, and then restart the deployment. Otherwise, a timeout error occurs when you start the deployment.
Python Uri
The Uniform Resource Identifier (URI) used to access the Python deployment file that you want to upload. Python deployment files can be .py files or .zip files.
Entry Module
Enter the entry point class of the Python application. If the Python deployment file that you select is a .py file, you do not need to configure this parameter. If the Python deployment file that you select is a .zip file, you must configure this parameter. For example, you can enter example.word_count in the Entry Module field.
Entry Point Main Arguments
The parameters of the deployment.
Python Libraries
A third-party Python package. The third-party Python package that you uploaded is added to PYTHONPATH of the Python worker process so that the package can be directly accessed in Python user-defined functions (UDFs). For more information about how to use third-party Python packages, see Use a third-party Python package.
Python Archives
Archive files. Only ZIP files such as .zip, .jar, .whl, and .egg are supported.
Archive files are decompressed to the working directory of the Python worker process. For example, if the name of the compressed file where the archive files are located is mydata.zip, the following code can be written in Python UDFs to access the mydata.zip archive file.
def map(): with open("mydata.zip/mydata/data.txt") as f: ...
For more information about Python Archives, see Use a custom Python virtual environment and Use data files.
Additional Dependencies
You can upload a file for the deployment, such as a Python deployment file and a data file that the deployment requires. For more information about Python dependencies, see Manage Python dependencies.
(Recommended) Select the dependency file that you uploaded.
You must upload a dependency file in advance. To upload a dependency file, click Upload Artifacts in the upper-left corner of the Artifacts page in the development console of Realtime Compute for Apache Flink. Alternatively, click the icon on the right side of Additional Dependencies in the Create Deployment dialog box. The dependency file that you upload is stored in the artifacts directory of the OSS bucket that you select to associate with your Realtime Compute for Apache Flink workspace when you activate the workspace. The file directory is in the format of
oss://<Name of the OSS bucket that is associated with your workspace>/artifacts/namespaces/<Name of the namespace>
.Enter the OSS bucket where the required dependency file is stored.
The OSS bucket to which the dependency file is uploaded must be the OSS bucket that you selected when you activate the current Realtime Compute for Apache Flink workspace.
Enter the URL of the required dependency file.
You must enter the URL of an external storage system that Realtime Compute for Apache Flink can access and is allowed to access. The access control list (ACL) of the external storage system must be public-read or Realtime Compute for Apache Flink must have the permission to access the external storage system. Only URLs that end with file names are supported, such as http://xxxxxx/<file>.
NoteThe dependency file that you upload by using one of the preceding methods can also be downloaded to the destination machine. When the deployment is running, the dependency file is loaded to the /flink/usrlib directory of the pod in which the JobManager and TaskManager reside.
If you select a session cluster for Deployment Target, you cannot configure the dependency file for the deployment.
Deployment Target
Select the desired queue or session cluster from the drop-down list. We recommend that you do not use session clusters in the production environment. For more information, see Manage queues and Step 1: Create a session cluster.
NoteMetrics of deployments that are deployed in session clusters cannot be displayed. Session clusters do not support the monitoring and alerting feature and the Autopilot feature. Session clusters are suitable for development and test environments. We recommend that you do not use session clusters in the production environment. For more information, see Debug a deployment.
Description
Optional. Enter the description for the deployment.
Label
After you specify labels for a deployment, you can search for the deployment by label key and label value on the Deployments page. You can specify a maximum of three labels for a deployment.
More Setting
If you turn on the switch, you must configure the following parameters:
Kerberos Name: Select a Hive cluster that supports Kerberos authentication from the drop-down list. For more information about how to create a Hive cluster that supports Kerberos authentication, see Register a Hive cluster that supports Kerberos authentication.
principal: a Kerberos principal, which can be a user or a service. A Kerberos principal is used to uniquely identify an identity in the Kerberos encryption system.
In the Create Deployment dialog box, click Deploy.
References
You can configure and modify a deployment and the resources of the deployment before you start the deployment or after you publish the draft for the deployment. For more information, see Configure a deployment and Configure resources for a deployment.
After you create a deployment, you can start the deployment on the Deployments page to run the deployment. For more information about how to start a deployment, see Start a deployment.