Realtime Compute for Apache Flink:October 23, 2023

Feature

Description

References

Optimized automatic tuning capabilities

The stable strategy is supported in Autopilot. The strategy helps the system identify the resource configuration that is suitable for the entire running cycle of a deployment. The system automatically adjusts the resource configuration of a deployment only when the system finds a resource configuration that is more suitable for the entire running cycle of the deployment. Otherwise, the system does not modify the existing resource configuration. This prevents the modification of the entire deployment resources due to changes in traffic bursts. This strategy helps the deployment run in a stable manner and reduces unnecessary changes and fluctuations to make the deployment reach the convergence state.

Configure automatic tuning

Scheduled tuning plans

Multiple scheduled tuning plans can be configured for a deployment, and the scheduled tuning plan in use can be changed without the need to cancel the deployment. One scheduled tuning plan can also contain multiple scheduled tuning strategies.

Configure automatic tuning

Data lineage of Flink SQL deployments

Data lineage of Flink SQL deployments can be viewed. You can view data lineage to find the Flink SQL deployment that uses a specific field in a table. You can also view data lineage to identify the field-level relationships between the source and result tables of the Flink SQL deployment. This helps you manage deployments and data assets in an efficient manner.

View data lineage

Dynamic updates of parameter configurations

The configuration of the Parallelism parameter and specific runtime parameters of a Flink deployment can be dynamically modified without the need to cancel the deployment.

Dynamically update the parameter configuration of a deployment

Optimization suggestions on SQL deployments

When you perform a syntax check on an SQL deployment, information about the risks and optimization suggestions on the SQL deployment is returned. You can optimize SQL statements based on the information about the risks and optimization suggestions.

Develop an SQL draft

Label-based search

When you create an SQL deployment, a JAR deployment, or a Python deployment, a label can be specified for the deployment. After you specify a label for the deployment, you can search for all deployments that use the label key or label value by label on the Deployments page in the console of fully managed Flink. This helps you manage deployments in an efficient manner.

Create a deployment

Enhanced deployment sorting and filtering capabilities

On the Deployments page in the console of fully managed Flink, deployments can be sorted by health score or business latency and can be filtered based on the user who modifies a deployment.

N/A

Logon authentication for RAM roles, members of a resource directory, and CloudSSO users

Realtime Compute for Apache Flink is fully adapted to the Alibaba Cloud account system. Alibaba Cloud accounts, RAM users, RAM roles, members of a resource directory, and CloudSSO users are supported. This simplifies user identity and access management and provides more comprehensive resource management and authorization mechanisms.

• Permission management

• Authorize an account to perform operations related to drafts and deployments

• Authorization in different logon methods

Complex data structures and exclusive Tunnel access supported by the MaxCompute connector

The MaxCompute connector can be used to read and write data of the JSON, ARRAY, MAP, and STRUCT types in MaxCompute. The MaxCompute connector also allows you to specify an exclusive Tunnel for service access.

MaxCompute connector

Support for the ALL cache policy when the ApsaraDB for Redis connector is used for a dimension table

When the ApsaraDB for Redis connector is used for a dimension table, the cache parameter can be set to ALL. This helps improve the processing performance.

ApsaraDB for Redis connector

ApsaraDB RDS for MySQL instances with transparent data encryption (TDE) enabled supported by MySQL catalogs

ApsaraDB RDS for MySQL instances for which TDE is enabled are supported by MySQL catalogs.

N/A

OSS-HDFS endpoint supported by the oss.endpoint parameter of a Data Lake Formation (DLF) catalog

The oss.endpoint parameter can be set to an OSS-HDFS endpoint for a DLF catalog.

• Hudi connector

• Manage DLF catalogs

Type normalization mode for the schema change in the CREATE TABLE AS statement when the StarRocks connector is used for a result table

The type normalization mode for the schema change is supported when you run a deployment that executes the CREATE TABLE AS statement to synchronize data and uses the StarRocks connector for a result table. In this case, if the length of a field of a specific data type in the source table changes but the length after the change is compatible with the length of the related field in the StarRocks result table, the synchronization is not affected.

N/A

Column of the DATE data type used as a partition key for a Hologres result table

If a column of the DATE data type is used as a partition key, a partitioned table can be automatically created.

Hologres connector

Data reading from and writing to ApsaraMQ for RocketMQ 5.x tables by using the ApsaraMQ for RocketMQ connector

The ApsaraMQ for RocketMQ connector can be used to read data from or write data to ApsaraMQ for RocketMQ 5.x tables.

ApsaraMQ for RocketMQ connector

Performance optimization for miniBatch and JOIN operations in specific scenarios

The performance of the miniBatch feature and JOIN operations is optimized. If you perform a JOIN operation after an operation such as Change Data Capture (CDC) or deduplication and aggregation is performed, the throughput in specific scenarios can be increased by about 130%.

N/A