Realtime Compute for Apache Flink:April 13, 2023

Feature

Description

References

Apache Paimon streaming data lake

The Apache Paimon streaming data lake is in invitational preview. Realtime Compute for Apache Flink supports data reading from and data writing to Apache Paimon based on Alibaba Cloud Object Storage Service (OSS).

Apache Paimon connector

Apache Paimon catalogs

Built-in Apache Paimon catalogs can be used together with Flink SQL to develop a Flink-based real-time data lake solution.

Manage Apache Paimon catalogs

Data writing to OSS-HDFS in streaming mode

Data can be written to OSS-HDFS in streaming mode. This allows you to use multiple types of data stores.

OSS connector

Adjustment of metrics

• CPU metrics

  • The JM CPU Usage metric is added.

  • The TM CPU Load and JM CPU Load metrics are removed.

    Note

    The TM CPU Load and JM CPU Load metrics cannot reflect CPU utilization. Only the JM CPU Usage and TM CPU Usage metrics are used to monitor CPU utilization.

• Connector metrics

  • The numRecordsOut and numRecordsOutPersecond metrics are added for the Elasticsearch connector to improve operational observability.

  • The Enumerator metric is added for the MySQL Change Data Capture (CDC) connector.

• Elasticsearch

• View metrics

• Discard or restore metrics

Tair result tables

Tair result tables are supported. This allows you to use multiple types of data stores.

-

Enhancement of EXPLAIN statements in streaming SQL

The EXPLAIN PLAN_ADVICE statement is supported to provide more detailed optimization suggestions.

None

Optimization of dynamic Flink complex event processing (CEP)

Conditions based on Groovy expressions are supported. The implementation of nondeterministic finite automaton (NFA) and SharedBuffer is optimized to reduce the number of times a timer is created and improve performance.

Definitions of rules in the JSON format in dynamic Flink CEP

Flink machine learning (ML)

Flink ML supported by the engine is in invitational preview and provides the real-time machine learning feature.

-

Optimization of the draft development process

The Draft Editor page is changed to the SQL Editor page to provide an SQL development platform, which optimizes the draft development process in the following aspects:

• Provides an isolation mechanism between drafts and deployments to prevent online deployments from being affected by SQL development. The development of JAR and Python drafts is naturally isolated from online JAR and Python deployments. The development of JAR and Python drafts does not require SQL components, such as user-defined functions (UDFs), connectors, and catalogs. Therefore, the isolation mechanism is not required for JAR and Python drafts.

• Optimizes the process of creating JAR and Python deployments. JAR and Python deployments can no longer be created on the SQL Editor page. You can directly create JAR and Python deployments on the Deployments page. You can find the JAR and Python drafts that are created on the Draft Editor page in the draft archive.

Develop an SQL draft

Optimization of the creation process of JAR and Python deployments

The process of creating JAR and Python deployments is optimized. You can click Create Deployment on the Deployments page to create a JAR or Python deployment.

• Create a JAR deployment

• Create a Python deployment

Optimization of the deployment startup process

The information about the resources of a deployment and the related Flink configurations are displayed on the Configuration tab of the Deployments page. You can adjust resource configurations without the need to cancel the deployment. You need to only specify a start offset when you start a deployment.

Start a deployment

Addition of the Catalogs page

The Catalogs page is added. The Apache Flink community does not recommend the use of temporary tables. To avoid repeated use of DDL statements, we recommend that you use catalogs to create SQL drafts. This version provides enhanced catalog capabilities to support the use of catalogs in SQL drafts and allow you to manage SQL drafts in an efficient manner.

None

Optimization of the SQL deployment debugging process

The debugging process of SQL deployments is optimized. If you want to debug an SQL deployment, you can select an existing session cluster for debugging instead of configuring a new session cluster. This way, you do not need to frequently change clusters during deployment debugging if a deployment has multiple versions.

Debug a deployment

Addition of the Security page

The user authorization and key hosting features are integrated into the Security page. You can complete settings related to platform security on the Security page. In earlier versions, the key hosting feature is provided on the Key Replacement tab.

Manage a key

Addition of the Connectors page

The Connectors page is provided to allow you to view the types and versions of connectors supported by different engine versions and manage custom connectors.

Manage custom connectors

Alerts for failed deployments,

sending of alert notifications by phone, and search of contacts

The monitoring and alerting feature is optimized in the following aspects:

• Allows you to set the Notification parameter to Phone to send alert notifications by phone and allows you to search for contacts.

• Supports alerts for failed deployments.

Configure alert rules

Logon by using a role account

A role account can be used to log on to the console of fully managed Flink. By default, owner permissions are used. You cannot configure permissions for the role account.

-

Network detection

An IP address or a domain name can be used to check whether the running environment of a fully managed Flink deployment is connected to the upstream and downstream systems.

None

Custom catalogs

Built-in catalogs are provided in the Realtime Compute for Apache Flink console. After you register metadata by using a catalog, you do not need to frequently use DDL statements to create temporary tables when you create an SQL draft. You can also create custom catalogs and use the JAR packages of the custom catalogs.

-

Enhanced intelligent deployment diagnostics capabilities

The intelligent deployment diagnostics feature is enhanced. This feature helps you analyze error logs that are generated during draft development and deployment running. When you view information on the log page, the system automatically analyzes the logs and provides executable operation suggestions.

Perform intelligent deployment diagnostics

Upgrade of the Log Service connector client

The performance and stability of the Log Service connector are improved.

None

Extended CEP SQL

The loop continuity declaration can be used together with the UNTIL syntax in CEP SQL.

None