All Products
Search
Document Center

Realtime Compute for Apache Flink:Activate fully managed Flink

Last Updated:Jan 10, 2024

This topic describes how to use an Alibaba Cloud account to activate fully managed Flink and the precautions that you need to take note of before you activate the service.

Prerequisites

  • You have an Alibaba Cloud account. If you do not have an Alibaba Cloud account, create one first.

  • The related permissions are granted to the Alibaba Cloud account. For more information, see Permission management.

  • The Object Storage Service (OSS) and Virtual Private Cloud (VPC) services are activated.

    • Fully managed Flink uses an architecture in which computing is decoupled from storage. This service uses OSS to store information, such as checkpoints, savepoints, logs, and JAR packages of deployments. If you have not activated OSS, activate it first. For more information, see Activate OSS.

    • For more information about how to activate VPC, see Create a VPC and a vSwitch.

Precautions

  • If you need to purchase more than 1,000 compute units (CUs) when you activate fully managed Flink, submit a ticket.

  • Application Real-Time Monitoring Service (ARMS) is automatically activated when you activate fully managed Flink. For more information about the billing rules of the involved cloud services, see Billable items.

  • When you select an OSS bucket, we recommend that you do not configure policies, such as the versioning and retention policies, for the OSS bucket that you select. The selected OSS bucket cannot be changed.

    If the versioning feature is enabled for your bucket, an excessive number of delete markers causes slow list processing. To avoid this issue, you must configure lifecycle rules to periodically remove delete markers and historical versions that you no longer use if you enable the versioning feature. This helps reduce storage costs and improves OSS performance. For more information about how to remove delete markers, see Configure lifecycle rules based on the last modified time for versioning-enabled buckets to reduce storage costs.

    Warning

    If you do not enable the versioning feature and you want to configure lifecycle rules, we recommend that you do not configure a deletion policy for the related directories of Flink, including artifacts, flink-jobs, flink-savepoints, sql-artifacts, plan, and flink-sessionclusters. If you configure a deletion policy for one of the preceding directories, Flink-related files may be deleted. As a result, Flink deployments cannot run as expected.

Procedure

  1. Log on to the Realtime Compute for Apache Flink console.

  2. In the Fully Managed Flink section, click Purchase.

  3. (Optional). Grant your Alibaba Cloud account the related permissions to access the services that fully managed Flink can call if you purchase fully managed Flink for the first time. On the Authorization Request page, click Authorize in RAM. On the Cloud Resource Access Authorization page, click Confirm Authorization Policy in the lower part of the page. After automated authorization is complete, your Alibaba Cloud account is granted the related permissions to access the services that fully managed Flink can call.

  4. On the buy page, enter the configuration information.

    Section

    Parameter

    Description

    Example

    Basic parameters

    Billing Method

    The Subscription and Pay-as-you-go billing methods are supported. For more information about billing details, see Billing items.

    Pay-as-you-go

    Duration

    You need to configure this parameter only when the Subscription billing method is used.

    You can select a subscription duration.

    -

    Auto-renewal

    You need to configure this parameter only when the Subscription billing method is used.

    After the auto-renewal feature is enabled, you can perform the following steps to modify the renewal policy: Log on to the Alibaba Cloud Management Console. In the top navigation bar, choose Expenses > Renewal Management.

    -

    CPU architecture

    You need to configure this parameter only when the Subscription billing method is used.

    You can set this parameter to Intel X86 Architecture or YITIAN ARM Architecture based on your business requirements.

    Note

    You can set this parameter to YITIAN ARM Architecture only for a subscription workspace that resides in Zone J in the China (Hangzhou) region. You can view the details on the buy page of the Realtime Compute for Apache Flink console.

    Intel X86 Architecture

    Region

    We recommend that you select the same region as the region in which upstream and downstream storage reside.

    China (Beijing)

    Zone

    We recommend that you select the same zone as the zone in which upstream and downstream storage reside.

    Zone F

    Network

    VPC

    We recommend that you select the same VPC in which the fully managed Flink workspace resides.

    • If the destination upstream and downstream storage resides in a VPC, the fully managed Flink workspace must be in the same VPC as the destination upstream and downstream storage.

    • Only the following CIDR blocks or their subsets are supported: 192.168.0.0/16, 172.16.0.0/12, and 10.0.0.0/8. These CIDR blocks are standard private CIDR blocks as defined by Request for Comments (RFC) documents.

    flink-test-vpc

    VSwitch

    An IP address is assigned to each TaskManager instance and each JobManager instance of a fully managed Flink deployment. You can select one to five vSwitches to properly plan the CIDR blocks based on the scale of fully managed Flink deployments.

    Note

    Only the vSwitches that are in the specified VPC and belong to the same zone as the fully managed Flink workspace are displayed in the console of fully managed Flink. If no available vSwitch is displayed, you can create a vSwitch.

    flinktest-vsw-2ze4fyq366itq6xqp****

    Workspace settings

    Workspace Name

    The name must be 1 to 60 characters in length and can contain only lowercase letters, digits, and hyphens (-). The name must start with a lowercase letter and cannot end with a hyphen (-).

    Important
    • After the fully managed Flink service is activated, you cannot change the value of Workspace Name.

    • Workspace names within the same Alibaba Cloud account must be unique. Do not enter the name of an existing workspace.

    flink-test

    Resource Quota

    You need to configure this parameter only when the Subscription billing method is used. Computing resources are billed based on the quota that is provided in this method.

    -

    Quota Ceiling

    You need to configure this parameter only when the Pay-as-you-go billing method is used. This parameter specifies the maximum number of compute units (CUs) that can be configured for a workspace. Default value: 1000. You can change the value of this parameter. The maximum value of this parameter is 9999999.

    Storage

    OSS Bucket

    An OSS bucket is used to store information, such as checkpoints, savepoints, logs, and JAR packages of deployments. The fully managed Flink service creates directories in the bucket that you select to store different types of data. By default, the data is retained for seven days.

    • artifacts: The uploaded JAR packages are stored in this directory.

    • flink-jobs: The high availability (HA) information and checkpoints of Flink deployments are stored in this directory.

    • flink-savepoints: If you click Savepoint in the console of fully managed Flink, savepoints are created for the deployment and the final savepoint file is stored in this directory.

    • logs: If you set Log Template to OSS for your deployment, the logs of your deployment are stored in this directory.

    • sql-artifacts: Files on which user-defined functions (UDFs) and connectors depend are stored in this directory.

    • plan: In expert mode, the configured resource information is stored in this directory.

    • flink-sessionclusters: The HA information and checkpoints of session clusters are stored in this directory.

    Important
    • After the fully managed Flink service is activated, you cannot change the value of OSS Bucket.

    • The OSS bucket must be in the same region as the fully managed Flink service.

    • For more information about how to select an OSS bucket, see Precautions.

    flink-test-oss

    Monitoring Service

    Monitoring Service

    After you activate the fully managed Flink service, ARMS is automatically activated to provide the commercial edition of Prometheus Service. For more information about the billing of the Prometheus Service, see Pay-as-you-go.

    Selected

  5. Click Confirm Order and complete the payment to activate the fully managed Flink service.

    After you complete the payment, click Console. On the Fully Managed Flink tab, you can view the workspace that is being created. In most cases, the workspace can be created within 5 to 10 minutes after you complete the payment.

  6. Perform operations based on your business requirements. The following table describes the operations that you can perform after fully managed Flink is activated.

    Operation

    Description

    View the details of a workspace

    On the Fully Managed Flink tab of the Realtime Compute for Apache Flink console, you can find the workspace that you want to manage and choose More > Workspace Details in the Actions column to view information about the workspace. The information includes the workspace name, workspace ID, OSS bucket, VPC name, VPC ID, and vSwitch.

    Create a namespace

    After a workspace is created, a namespace named Workspace name-default is automatically generated.

    You can create a namespace. For more information, see Create and manage a namespace.

    Develop drafts and maintain deployments

    On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column to go to the console of fully managed Flink. You can develop drafts and maintain deployments. For more information about related operations, see Draft development and O&M Management.

    Fully managed Flink provides various upstream and downstream built-in connectors. For more information, see Supported connectors.

    Note

    If you want to allow other Alibaba Cloud accounts or RAM users to use your resources in fully managed Flink, you must grant them the required permissions, such as the permissions to develop a draft. For more information, see Authorize an account to perform operations related to drafts and deployments and Grant permissions to a RAM user.

    Renew a workspace

    Before the workspace expires, you can renew a subscription workspace to extend the validity period of the workspace. For more information, see Renewal policy.

    Reconfigure workspace resources

    You can change the resource configuration of workspaces and namespaces. For more information, see Reconfigure resources.

    Switch the billing method

    You can switch the billing method of a workspace between pay-as-you-go and subscription in the Realtime Compute for Apache Flink console. For more information, see Switch between billing methods.

    Change a vSwitch

    You can change the vSwitch to resolve deployment startup failures that are caused by insufficient IP addresses of the vSwitch that you select. For more information, see Modify a vSwitch.

    Release resources

    On the Fully Managed Flink tab of the Realtime Compute for Apache Flink console, you can find the workspace that you want to manage and choose More > Release Resources in the Actions column to release pay-as-you-go resources. You are not charged for the workspace after it is released. The ARMS service is released when you release the fully managed Flink workspace.