Fully managed Flink allows you to edit and run jobs by using SQL code. This topic describes how to perform various operations, such as activating fully managed Flink, creating SQL jobs, and running SQL jobs.
- An Alibaba Cloud account is created and has a sufficient balance.
- For more information about how to create an Alibaba Cloud account, go to the Sign up to Alibaba Cloud page page.
- The Alibaba Cloud account has a balance of at least USD 100 or a voucher or coupon of the equivalent value.
- A role is assigned to the Alibaba Cloud account. For more information, see Assign a role to an Alibaba Cloud account.
- Step 1: Create a fully managed Flink instanceCreate a fully managed Flink instance that uses the pay-as-you-go billing method in the China (Beijing) region.
- Step 2: Create an SQL job and write business codeOn the Draft Editor page, create an SQL streaming job and write DDL and DML code of the job.
- Step 3: Start the job and view the computing resultsOn the Developments page, start the job and view the job status.
Step 1: Create a fully managed Flink instance
- Log on to the Realtime Compute for Apache Flink console.
- In the Fully Managed Flink section, click Purchase.
- On the buy page, enter the configuration information.
Section Parameter Example value Description Basic Configurations Billing Method Pay-as-you-go The Subscription and Pay-as-you-go billing methods are supported. The region where fully managed Flink is supported. Beijing The Singapore (Singapore) region is supported.Note We recommend that you select the same region as the upstream and downstream storage. Zone Zone F We recommend that you select the same zone as the upstream and downstream storage. Network SLB Service - The SLB service is selected and activated by default. VPC flink-test-vpc We recommend that you select the same VPC as the upstream and downstream storage. vSwitch flinktest-vsw-2ze4fyq366itq6xqp**** An IP address is assigned to each TaskManager instance and each JobManager instance of a Flink job. You can select one to five vSwitches to properly plan the Classless Inter-Domain Routing (CIDR) blocks based on the scale of Flink jobs. Workspace Workspace Name flink-test The name must be 1 to 60 characters in length and can contain letters, digits, and hyphens (-). It must start with a letter. Storage OSS Bucket flink-test-oss The OSS bucket is used to store job information, such as checkpoints, logs, and JAR packages. The fully managed Flink service creates the following directories in the bucket that you select to store different types of data:
- artifacts: The uploaded JAR files are stored in this directory.
- flink-jobs: The high availability (HA) information and checkpoints of Flink jobs are stored in this directory.
- flink-savepoints: If you click Savepoint in the console of fully managed Flink, the savepoint operation is triggered and the final savepoint file is stored in this directory.
- logs: If you set Log Template to OSS for your job, the logs of your job are stored in this directory.
- sql-artifacts: Files on which user-defined functions (UDFs) and connectors depend are stored in this directory.
- plan: In Expert mode, the configured resource information is stored in this directory.
- flink-sessionclusters: The HA information and checkpoints of session clusters are stored in this directory.
- After the fully managed Flink service is activated, OSS Bucket cannot be changed.
- The OSS bucket must be in the same region as the fully managed Flink service.
- For more information about how to select an OSS bucket, see Usage notes.
Monitoring Monitoring Service - Prometheus Service is selected and activated by default.
- Click Confirm Order and complete the payment to activate the fully managed Flink service. Note After you complete the payment, click Console. On the Fully Managed Flink tab, you can view the workspace that is being created. In most cases, the workspace can be created in 5 to 10 minutes after you complete the payment.
Step 2: Create an SQL job and write business code
- Create an SQL job.
- Log on to the Realtime Compute for Apache Flink console.
- On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.
- In the left-side navigation pane, click Draft Editor.
- Click New.
- In the New Draft dialog box, configure the parameters of the job. The following table describes the
Parameter Example value Description Name flink-test The name of the job.Note The job name must be unique in the current project. Type STREAM / SQL Streaming jobs and batch jobs support the following file types:
Note Ververica Platform (VVP) 2.4.1 and its later versions and Ververica Runtime (VVR) 3.0.1 and its later versions support batch jobs.
Deployment Target vvp-workload The name of the Flink cluster in which you want to deploy the job. Fully managed Flink supports two cluster types: per-job cluster and session cluster. For more information about the differences between the two types of clusters, see Configure a development and test environment (session cluster). Locate Development The folder in which the code file of the job is saved. By default, the code file of the job is stored in the Development folder.
You can also click the icon next to an existing folder to create a subfolder.
- Click OK.
- Copy the following job code to the code editor. Create the datagen source table that generates random data, and insert the data in the name field in the datagen source table into a blackhole result table. The following example shows the job code.
CREATE TEMPORARY TABLE datagen_source( randstr VARCHAR ) WITH ( 'connector' = 'datagen' ); CREATE TEMPORARY TABLE print_table( randstr VARCHAR ) WITH ( 'connector' = 'print'', 'logger' = 'true' ); INSERT INTO print_table SELECT SUBSTRING(randstr,0,8) from datagen_source;
- Click Validate to check the syntax.
- After the verification succeeds, click Publish.
Step 3: Start the job and view the computing results
- In the left-side navigation pane, click Deployments.
- Find the job that you want to start and click Start in the Actions column.
- In the Deployment Starting Configuration dialog box, click Confirm Running. After you click Confirm Running, you can view the transition process from a current state to a desired state and the final result. When the state changes to RUNNING, the job is running properly.
- On the Deployments page, click the name of the job that is started. On the Task Manager tab, view the computing results in logs.
- For more information about how to create a project or modify the configurations of project resources, see Manage projects.
- For more information about how to debug a job to improve the resource utilization of JobManager, see Configure a development and test environment (session cluster).
- For more information about how to enable the Autopilot feature, see Configure Autopilot.
- For more information about how to configure the monitoring and alerting feature, see Configure alert rules in the console of fully managed Flink.