VS Code local job development - Realtime Compute for Apache Flink

The Realtime Compute for Apache Flink extension for VS Code streamlines the local development process for Flink jobs. You can use the extension to develop, deploy, and publish SQL, JAR, and Python jobs from your local machine. You can also sync job configurations from your online environment. This topic describes how to use the VS Code extension.

Prerequisites

You have activated Realtime Compute for Apache Flink.
You have obtained an AccessKey ID and an AccessKey secret. The identity that is associated with the AccessKey must have the required permissions for the target project. For more information, see How do I find my AccessKey ID and AccessKey secret? and Authorize users in the development console.
VS Code editor v1.74.0 or later is installed. You can download it from the VS Code official website.

Limits

Log configuration is not supported.
Pulling existing online jobs is not supported.

Install and configure the extension

Download and install the extension.
- Method 1: Go to the VSCode Marketplace and click Install.
- Method 2:
  1. Open VS Code and click EXTENSIONS in the navigation pane on the left.
  2. In the search bar, search for the Flink Realtime Compute Studio extension.
  3. Click Install for the target extension.
In the menu bar, click File > Open Folder and select a local folder.

Configure the AccessKey information.

Apply to the current workspace (current file directory)

Click in the navigation pane on the left. In the Configure AK&SK section, click Configure in a .vvprc file. The extension automatically creates a file with the .vvprc extension.

Enter the following configuration information.

{
  "accessKey": "xxx",
  "secretKey": "xxx",
  "namespace": "xxx",
  "workspace": "xxx",
  "regionId" : "xxx"
}

Parameter	Description	Notes
accessKey	The AccessKey ID of your Alibaba Cloud account.	For more information, see How do I find my AccessKey ID and AccessKey secret?
secretKey	The AccessKey secret of your Alibaba Cloud account.
namespace	The name of the Flink project.	For more information, see View and delete a project.
workspace	The ID of the Flink workspace.	For more information, see How do I view information such as the workspace ID?.
regionId	The region ID.	See the region ID in Endpoints.

Apply globally

Click in the navigation pane on the left.
In the Configure AK&SK section, click Configure in global user settings. You are redirected to the global settings of VS Code.

Select User or Workspace to configure the settings as needed.

Note

The User configuration applies to all projects and workspaces in the VS Code editor and is stored in the global configuration file. The Workspace configuration applies only to the currently open workspace.

Parameter	Description	Notes
Access Key	The AccessKey ID of your Alibaba Cloud account.	For more information, see How do I find my AccessKey ID and AccessKey secret?
Secret Key	The AccessKey secret of your Alibaba Cloud account.	For more information, see How do I find my AccessKey ID and AccessKey secret?
Namespace	The name of the Flink project.	For more information, see View and delete a project.
Workspace	The ID of the Flink workspace.	For more information, see How do I view information such as the workspace ID?.
Region ID	The region ID.	See the region ID in Endpoints.

Note

Configuration priority: Current folder configuration > Parent folder configuration > Global configuration.

After the configuration is applied, the name of the corresponding project appears in the status bar at the bottom of the job editor.

(Optional) You can install the following extensions to improve your coding efficiency.
- Flink SQL: Install the Flink SQL Language Features extension. This extension provides Flink SQL language services and supports the complete Flink SQL syntax.
- Flink configuration: Install the YAML VS Code extension. This extension provides an auto-complete feature for Flink configurations.

Procedure

Step 1: Create a job

Method 1: Click in the navigation pane on the left. In the QUICK START section, click Create a new deployment. Enter a name at the top and press Enter to create a file with the .vvp extension. The extension generates a job with default fields.
Method 2: In the file tree, create a file with the .vvp extension. The extension generates a job with default fields.

Step 2: Configure the job

Configure parameters in the UI form.

If you are familiar with job configuration, you can click the Raw File button in the upper-left corner to configure the job in the source file. In the UI form, you can click a parameter name to navigate to the corresponding location in the source file.

Basic: Basic parameter settings

SQL job

Parameter	Description
Deployment Name	The name of the job.
Execution Mode	The deployment mode. Valid values: STREAM and BATCH.
Engine Version	For more information about database engine versions, see Database engine versions. Use the recommended version.
Deployment Target	The destination resource queue for the deployment. For more information, see Manage resource queues.
Deployment Type	Select SQL.
SQL Script	Click Open in external editor to open the local editor. Write DDL and DML code. Note The editor opened from here is a virtual file, not a file in the actual file system. Because of this, if you reopen VS Code while the virtual file is still open, the editor window displays the `The editor could not be opened because the file was not found` error message. To fix this, click Open in external editor again.
Additional Dependencies	The URI of the additional dependencies required for the job.
Description	Optional. Enter remarks.

JAR job

For more information about parameter settings, see Deploy a JAR job.

Parameter	Description
DeploymentName	The name of the job.
Execution Mode	The deployment mode. Valid values: STREAM and BATCH.
Engine Version	The database engine version. For more information, see Database engine versions. Use the recommended version.
Deployment Target	The destination for the deployment. Select a destination resource queue. For more information, see Manage resource queues.
Deployment Type	The type of job to deploy. Select JAR.
JAR URI	The OSS path or URL of the JAR package. Make sure you have uploaded the dependency files to Resource Management in the Flink development console or to a bound OSS Bucket and obtained the corresponding path.
Entry Point Class	The entry class of the program. If your JAR package does not specify a main class, enter the standard path of your entry point class.
Entry Point Main Arguments	Enter the input parameters to be invoked in the main method.
Additional Dependencies	The OSS path or URL of the destination additional dependency files. Only URLs that end with a file name are supported. Make sure you have uploaded the dependency files to Resource Management in the Flink development console or to a bound OSS Bucket and obtained the corresponding path.
Description	Optional. Enter remarks.

Python job

For more information about parameter settings, see Deploy a Python job.

Parameter	Description
Deployment Name	The name of the job.
Execution Mode	The deployment mode. Valid values: STREAM and BATCH.
Engine Version	The database engine version. For more information, see Database engine versions. Use the recommended version.
Deployment Target	The destination for the deployment. Select a destination resource queue. For more information, see Manage resource queues.
Deployment Type	The type of job to deploy. Select Python.
Python Uri	The address of the Python job file. The file can be a .py file or a .zip file.
Entry Module	The entry class for the program. If the Python job file is a .py file, you do not need to specify this parameter. If the Python job file is a .zip file, enter the entry module.
Entry Point Main Arguments	Enter the input parameters to be invoked in the main method.
Python Libraries	Third-party Python packages. Third-party Python packages are added to the PYTHONPATH of the Python worker process so that they can be directly accessed in Python user-defined functions. For more information about how to use third-party Python packages, see Use third-party Python packages.
Python Archives	Archive files. For more information, see Use a custom Python virtual environment and Use data files.
Additional Dependencies	The OSS path or URL of the destination additional dependency files. Make sure you have uploaded the dependency files to Resource Management in the Flink development console or to a bound OSS Bucket and obtained the corresponding path.
Description	Optional. Enter remarks.

Resources: For more information about job resource parameters, see Configure job resources.
Parameters: For more information about job running parameters, see Configure running parameters.

Step 3: Deploy the job

After you configure the job, click Create at the bottom of the page to deploy it as a new job. For an existing local job, click Update to deploy your changes.

For SQL jobs, the extension validates the SQL Script, Resource Plan, and other job configurations, and then displays the validation results.

Step 4: Start the job

For a deployed job, the latest status is displayed in the upper-right corner. You can click Start and select a startup state. For more information about startup states, see Start a job.

Step 5: Perform O&M on the job

Open Online: To view the detailed running status of the job or perform O&M operations, you can click Open Online in the upper-left corner. You are redirected to the O&M page in the Flink development console.
Synchronize: This option synchronizes jobs from your online environment. Click Synchronize to open the online job preview page. The preview page does not show draft updates that are made to SQL jobs in the console. After you confirm that the preview is correct, click the Accept button in the upper-right corner. The extension then overwrites the local job file with the online job.
Cancel: Terminate the job.

References

For more information about supported connectors, see Supported connectors.
For more information about how to develop and debug Flink JAR jobs, see Develop JAR jobs.
For more information about how to develop and debug Flink Python jobs, see Develop Python jobs.
To improve the performance of Flink SQL jobs, see High-performance Flink SQL optimization techniques.