Realtime Compute for Apache Flink:Feature overview

Category

Feature

Description

Reference

Development

Developing SQL drafts

When you write code for an SQL draft in the Realtime Compute for Apache Flink console, you can use various built-in connectors provided by the service, user-defined functions (UDFs), and custom connectors.

• Develop an SQL draft
• Getting started with an SQL deployment

Developing a PyFlink program for Python deployment

You can develop your Python program in an on-premises environment, and deploy it and start the deployment in the development console of Realtime Compute for Apache Flink.

Developing and packaging a Flink program for JAR deployment

Realtime Compute for Apache Flink DataStream provides flexible programming models and APIs, which allow you to specify data conversions, perform operations, and configure operators to meet requirements for complex business logic and data processing.

Scripts

A script can contain the CALL command and DDL, DQL, and DML statements. You can use scripts to create and manage catalogs and tables, perform data queries, and manage Apache Paimon tables. You can use the Explain statement in a script to view execution plans and troubleshoot issues.

Function management

You can use a UDF in developing an SQL draft.

Draft debugging

You can enable the debugging feature to simulate deployment running, check outputs, and verify the business logic of SELECT and INSERT statements. This feature improves the development efficiency and reduces the risks of poor data quality.

Data access

Connectors

Realtime Compute for Apache Flink provides a variety of built-in connectors to read data from upstream data stores, write data to downstream data stores, and synchronize data. Realtime Compute for Apache Flink also allows you to upload and use custom connectors.

• Connectors
• Supported connectors

Data formats

Realtime Compute for Apache Flink supports multiple data formats that can be used with connectors. A data format defines how to map binary data onto table columns.

Metadata Management

Built-in catalogs

After you create a catalog, you can directly read metadata in the development console of Realtime Compute for Apache Flink without the need to manually register tables. This improves data development efficiency and data accuracy.

• Manage catalogs
• Ingest data into data warehouses in real time
• Ingest log data into data warehouses in real time
• Getting started with real-time data ingestion into data lakes based on Apache Paimon

Custom catalogs

If built-in catalogs cannot meet your business requirements, you can use custom catalogs.

Category

Feature

Description

Reference

Deployment management

Deployment creation

Deployment effectively isolates development from production. Creating a deployment does not affect existing, running jobs. A job is officially submitted after the deployment is started or restarted.

Deployment configuration

You must configure a deployment before starting it.

Dynamic parameter update

Dynamic update of the parameter configuration of a Flink deployment can make the parameter configuration take effect more quickly. This helps reduce the service interruption time caused by deployment startup and cancellation and facilitates dynamic scaling of TaskManagers and checkpoint-based troubleshooting.

Start a deployment

After you develop and deploy a draft, you must start the deployment on the Deployments page to run the deployment.

Job logs

If an exception occurs at deployment startup or at runtime, you can view the logs for troubleshooting.

• View the exception logs of a deployment
• View the boot logs and operational logs of a deployment
• View the logs of a historical job

State management

The checkpoints and savepoints of a deployment are collectively called its state set, which is used to manage the lifecycle of the deployment.

Deployment diagnostics and optimization

The intelligent deployment diagnostics feature is used to monitor the health status of a deployment, analyze and diagnose exception logs, exceptions, and risks of a deployment, and provide optimization suggestions based on the diagnostic results. Flink deployments support two automatic tuning modes to allocate resources: Autopilot and scheduled tuning.

Data lineage

You can use the data lineage feature to track data in a deployment. This way, you can efficiently manage and optimize data streams, quickly identify issues, and evaluate the impact of the issues.

Monitoring and alerting

Realtime Compute for Apache Flink allows you to use CloudMonitor (a free monitoring service) or Application Real-Time Monitoring Service (ARMS) to implement deployment monitoring and alerting. You can configure metric alert rules or subscribe to event-triggered alerts, detecting and handling exceptions at the earliest opportunity.

Workflows

Workflow management

A workflow is a directed acyclic graph (DAG) that you can create by dragging tasks and establishing associations among tasks. If you want to run tasks at specific points in time, you can create a workflow and configure tasks and scheduling policies in the workflow.

Workflow instance management

After you create a workflow, you can perform operations on existing workflow instances and task instances.

Resources

Resources

Realtime Compute for Apache Flink creates directories in the Object Storage Service (OSS) bucket that is associated with your Realtime Compute for Apache Flink workspace when you purchase the workspace. The directories are used to store data files (checkpoints, savepoints, logs, and JAR packages, etc.) that are required for running a deployment or generated at run time.

Queue management

Queue management

You can isolate and manage resources through queue management.

Category

Feature

Description

Reference

Security center

Hive Kerberos

You can register a Hive cluster that supports Kerberos authentication in the console to allow Realtime Compute for Apache Flink deployments to access the Hive cluster.

Key hosting

Realtime Compute for Apache Flink allows you to configure variables and keys to prevent security risks that are caused by the use of information such as plaintext AccessKey pairs and passwords. You can reuse variables and keys to avoid repeatedly writing the same code or values and simplify configuration management. You can reference a variable or a key in various scenarios, such as the development of SQL, JAR, or Python drafts, log output configuration, and UI-based parameter configuration.

Permission management

Member management

To allow multiple users to collaborate in a namespace in the development console of Realtime Compute for Apache Flink, such as performing SQL development and O&M, you can add members to the namespace and assign roles that have predefined permissions to the members.

Role management

You can create custom roles to grant permissions based on your business requirements. This improves the flexibility of permission management and ensures security.

Category

Feature

Description

Reference

Namespace management

Namespace management

A namespace is a basic unit for managing deployments in Realtime Compute for Apache Flink. Configurations, deployments, and permissions of each namespace are separately managed. You can create multiple namespaces and assign separate resources and permissions to each namespace. This way, the resources and permissions of multiple tenants are isolated among namespaces.

Resource tag management

A tag consists of a tag key and a tag value. Tags are used to identify cloud resources. Tags allow you to categorize, search for, and aggregate cloud resources that have the same characteristics from different dimensions. This helps you manage cloud resources in an efficient manner.