Realtime Compute for Apache Flink:Version 2023-04-13

Feature

Details

References

Apache Paimon streaming data lake support

As a preview feature, the engine supports reading from and writing to Paimon on Alibaba Cloud Object Storage Service (OSS).

Paimon

Apache Paimon Catalog support

Use the built-in Apache Paimon Catalog to build real-time data lake solutions with Flink SQL.

Manage Paimon Catalog

OSS-HDFS streaming write support

Write streaming data directly to OSS-HDFS to expand data storage options.

Object Storage Service (OSS)

Monitoring metric updates

• CPU metric changes

  • Add JM CPU Usage metric.

  • Remove TM CPU Load and JM CPU Load metrics.

    Note

    CPU Load does not reflect CPU usage. CPU Usage will be the sole monitoring metric for CPU utilization.

• Connector metric changes

  • Add numRecordsOut and numRecordsOutPerSecond metrics to the Elasticsearch connector to improve observability.

  • Add Enumerator metrics to the MySQL CDC connector.

• Elasticsearch

• View monitoring metrics

• Enable or disable monitoring metrics

Tair sink table support

Add Tair as a sink table option to expand data storage choices.

-

Stream SQL EXPLAIN enhancement

Add plan_advice mode to provide detailed optimization suggestions.

None

Dynamic CEP optimization

Support Groovy expressions for conditions. Optimize NFA and SharedBuffer implementations. Reduce timer creation and improve performance.

JSON format definition of rules in dynamic CEP

Flink ML support

As a preview feature, the engine supports Flink ML to simplify real-time machine learning.

-

Job development workflow optimization

Optimize the job development page into an SQL development page. This page provides a dedicated SQL development platform:

• Add deployment isolation to prevent SQL development from affecting production jobs. Jar and Python jobs are naturally isolated from production. They do not use UDFs, connectors, or catalogs from SQL, so they do not need deployment isolation.

• Optimize Jar and Python job creation. The SQL development page no longer supports creating Jar and Python jobs. Deploy them directly from the job O&M page. Draft Jar and Python jobs created earlier in the development interface remain available in the job archive.

Job Development

Jar and Python job deployment optimization

Optimize the creation flow. Deploy Jar and Python jobs directly from the job O&M page.

• Deploy JAR Jobs

• Deploy Python Jobs (merged into Deploy Jobs; international site English localization pending before deprecation and relocation)

Job startup workflow optimization

Move resource and Flink Conf configurations to the deployment details page. Adjust resources without stopping the job. During startup, set only the start offset.

Start a Job

New metadata page

The Flink community discourages temporary tables. Use catalogs instead to create SQL jobs and avoid repeated DDL statements. This version enhances metadata capabilities. Use metadata more often in SQL jobs to simplify management.

None

SQL job debugging workflow optimization

Optimize SQL job debugging. You no longer need to configure a debug session cluster. Instead, select the session cluster during debugging. This avoids frequent cluster switching when multiple job versions exist.

Debug a Job

New security page

Combine user authorization and key hosting (previously variable substitution) into a single security page. Configure all platform security settings here.

Manage Variables

New connector page

Display connector jobs on a dedicated page. View supported connector types and versions per engine. Manage custom connectors.

Manage Custom Connectors

Job failure alerts

with phone alerts and contact search

Upgrade monitoring and alerting:

• Configure phone alerts. Search for contacts.

• Alerting for failed jobs is now supported.

Configure Monitoring and Alerts

Role-based account logon

Log on to the Flink console using a role-based account. Role accounts cannot have permissions assigned. They use owner permissions by default.

-

Network connectivity testing

Test network connectivity between your Flink job environment and upstream or downstream systems by entering an IP address or domain name.

None

Custom catalog support

Register metadata through catalogs. Avoid repetitive DDL statements to create temporary tables when building SQL jobs. The Realtime Compute development console includes built-in catalogs. You can also register and use custom catalog JARs.

-

Enhanced intelligent diagnostics

Analyze error logs from job development and runtime. When viewing logs, the system automatically analyzes content and suggests actionable steps.

Intelligent Job Diagnostics

SLS connector client upgrade

Improve SLS connector performance and stability.

None

CEP SQL extension

CEP SQL supports loop continuity declarations with UNTIL syntax.

None