Fully managed Flink allows you to edit and run jobs by using SQL code. This topic describes how to perform various operations. For example, you can activate fully managed Flink, create SQL jobs, and run SQL jobs.

Prerequisites

  • An Alibaba Cloud account is created and the account balance is sufficient.
    • For more information about how to create an Alibaba Cloud account, go to the Sign up to Alibaba Cloud page page.
    • The Alibaba Cloud account has a balance of at least USD 100 or a voucher or coupon of the equivalent value.
  • A role is assigned to the Alibaba Cloud account. For more information, see Assign a role to an Alibaba Cloud account.

Procedure

  1. Step 1: Create a fully managed Flink instance
    Create a fully managed Flink instance that uses the pay-as-you-go billing method in the China (Beijing) region.
  2. Step 2: Create an SQL job and write business code
    On the Draft Editor page, create an SQL streaming job and write DDL and DML code of the job.
  3. Step 3: Start the job and view the computing results
    On the Developments page, start the job and view the job status.

Step 1: Create a fully managed Flink instance

  1. Log on to the Realtime Compute for Apache Flink console.
  2. In the Fully Managed Flink section, click Purchase.
  3. On the buy page, configure the parameters.
    Activation
    Section Parameter Example value Description
    Basic configurations Billing Method Pay-as-you-go The Subscription and Pay-as-you-go billing methods are supported.
    Region China (Beijing) The Singapore (Singapore) region is supported.
    Note We recommend that you select the same region as the region in which upstream and downstream storage reside.
    Zone Zone F We recommend that you select the same zone as the zone in which upstream and downstream storage reside.
    Network SLB Service - The SLB service is selected and activated by default.
    VPC flink-test-vpc We recommend that you select the same virtual private cloud (VPC) as the upstream and downstream storage.
    vSwitch flinktest-vsw-2ze4fyq366itq6xqp**** An IP address is assigned to each TaskManager instance and each JobManager instance of a fully managed Flink job. You can select one to five vSwitches to properly plan the CIDR blocks based on the scale of fully managed Flink jobs.
    Workspace settings Workspace Name flink-test The name must be 1 to 60 characters in length and can contain letters, digits, and hyphens (-). The name must start with a letter.
    Storage OSS Bucket flink-test-oss The OSS bucket is used to store job information, such as checkpoints, logs, and JAR packages. The fully managed Flink service creates the following directories in the bucket that you select to store different types of data:
    • artifacts: The uploaded JAR files are stored in this directory.
    • flink-jobs: The high availability (HA) information and checkpoints of fully managed Flink jobs are stored in this directory.
    • flink-savepoints: If you click Savepoint in the console of fully managed Flink, the savepoint operation is triggered and the final savepoint file is stored in this directory.
    • logs: If you set Log Template to OSS for your job, the logs of your job are stored in this directory.
    • sql-artifacts: Files on which user-defined functions (UDFs) and connectors depend are stored in this directory.
    • plan: In Expert mode, the configured resource information is stored in this directory.
    • flink-sessionclusters: The HA information and checkpoints of session clusters are stored in this directory.
    Note
    • After the fully managed Flink service is activated, OSS Bucket cannot be changed.
    • The OSS bucket must be in the same region as the fully managed Flink service.
    • For more information about how to select a bucket, see Usage notes.
    Monitoring Monitoring Service - Prometheus Service is selected and activated by default.
  4. Click Confirm Order and complete the payment to activate the fully managed Flink service.
    Note After you complete the payment, click Console. On the Fully Managed Flink tab, you can view the workspace that is being created. In most cases, the workspace can be created in 5 to 10 minutes after you complete the payment.

Step 2: Create an SQL job and write business code

  1. Create an SQL job.
    1. Log on to the Realtime Compute for Apache Flink console.
    2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.
    3. In the left-side navigation pane, click Draft Editor.
    4. Click New.
    5. In the New Draft dialog box, configure the parameters of the job. The following table describes the parameters.
      Parameter Example value Description
      Name flink-test The name of the job.
      Note The job name must be unique in the current project.
      Type STREAM / SQL Streaming jobs and batch jobs support the following file types:
      • SQL
      • JAR
      • PYTHON
      Note Batch jobs are supported in Ververica Platform (VVP) 2.4.1 and later minor versions and Ververica Runtime (VVR) 3.0.1 and later minor versions.
      Deployment Target vvp-workload The name of the Flink cluster in which you want to deploy the job. Fully managed Flink supports two types of clusters: per-job cluster and session cluster. For more information about the differences between the two types of clusters, see Configure a development and test environment (session cluster).
      Location Development The folder in which the code file of the job is saved. By default, the code file of the job is stored in the Development folder.

      You can click the Create Folder icon to the right of an existing folder to create a subfolder.

    6. Click OK.
  2. Copy the following job code to the code editor.
    Create the datagen source table that generates random data, and insert the data in the name field in the datagen source table into a blackhole result table. The following example shows the job code.
    CREATE TEMPORARY TABLE datagen_source(
      randstr VARCHAR
    ) WITH (
      'connector' = 'datagen'
    );
    
    CREATE TEMPORARY TABLE print_table(
      randstr  VARCHAR
    ) WITH (
      'connector' = 'print',
      'logger' = 'true'
    );
    
    INSERT INTO print_table
    SELECT SUBSTRING(randstr,0,8) from datagen_source;
  3. Click Validate to check the syntax.
  4. After the verification succeeds, click Publish.

Step 3: Start the job and view the computing results

  1. In the left-side navigation pane, click Deployments.
  2. Find the job that you want to start and click Start in the Actions column.
    Start a job
  3. In the Deployment Starting Configuration dialog box, click Confirm Running.
    After you click Confirm Running, you can view the transition process from a current state to a desired state and the final result. When the state changes to RUNNING, the job is running as expected. Status transition
  4. On the Deployments page, click the name of the job that is started. On the Task Manager tab, view the computing results in logs.
    Result

References