All Products
Search
Document Center

MaxCompute:Release notes for key features

Last Updated:Oct 23, 2025

This topic describes the release notes for key features of MaxCompute.

For more information about the major feature updates of MaxCompute, see Product Updates.

2025

July

Feature

Description

Release date

Regions

References

MaxCompute supports automatic table partitioning based on time calculation functions

MaxCompute can automatically partition a table by applying specific functions to its time or date columns. The results of these functions are used to generate partition values.

2025-07-31

All regions

Automatic partition table based on time computing function

MaxCompute supports automatic table partitioning based on data write time

MaxCompute automatically retrieves the time when data is written to a table. It uses this time with a specified function, such as TRUNC_TIME, to generate partition column values and partition the table.

2025-07-31

All regions

Auto partition table based on data writing time

MaxFrame jobs support worker-level CPU and memory monitoring

MaxFrame lets you use Fuxi Sensor to view the actual CPU and memory consumption of a specific worker.

2025-07-17

All regions

Fuxi Sensor

New built-in large models added to MaxFrame AI Function

The MaxFrame AI Function adds built-in Qwen 3 series models, from Qwen3-0.6B to Qwen3-14B.

2025-07-17

All regions

MaxFrame AI Function

MaxFrame supports the partial commit feature for jobs

In large-scale jobs, some instances may run for a long time due to issues such as data skew or infinite loops. This can slow down the job or cause it to fail. The partial commit feature allows a small part of the job to be ignored, letting you retrieve results from successful instances even if the overall job fails.

2025-07-17

All regions

MaxFrame partial job submission function

May

Feature

Description

Release date

Regions

References

MaxCompute SQL engine enhancements

The new SQL V51 version is online:

  • New feature: Support for auto-partitioned tables based on time calculation functions. Use time functions to truncate time or date columns and generate values for partition key columns.

  • Syntax enhancements: The STRUCT syntax is enhanced to support the STRUCT(*) syntax. In addition, SHOW HISTORY FOR TASK now supports specifying a start position and length.

  • Function enhancements: Support for new functions, such as BOOL_AND, BOOL_OR, BITWISE_XOR_AGG, and BIT_COUNT.

  • Performance optimizations: Enhanced Shuffle Removal capability improves job performance when writing to clustered tables.

2025-05-30

All regions

Data warehouse engine

April

Feature

Description

Release date

Regions

References

Upgrade of MaxCompute data transmission service observability

Data accuracy is improved by optimizing the metric data collection and processing pipeline. New capabilities are added to support viewing resource usage at the project level, along with more usage options. This enhancement enriches the observability of the data transmission service, enabling support for a broader range of analysis scenarios and requirements.

2025-04-24

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China East 1 Finance, China (Hong Kong), Singapore, Japan (Tokyo), Malaysia (Kuala Lumpur), Indonesia (Jakarta), Germany (Frankfurt), UK (London), US (Silicon Valley), US (Virginia), and SAU (Riyadh - Partner Region)

Use resource observation

MaxFrame AI Function released

The MaxFrame AI Function feature allows users to call large models through a simple and user-friendly programming interface to perform offline processing on massive datasets within MaxCompute tables.

2025-04-09

China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Ulanqab), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Germany (Frankfurt), UK (London), US (Silicon Valley), US (Virginia), UAE (Dubai), and SAU (Riyadh - Partner Region)

MaxFrame AI Function

March

Feature

Description

Release date

Regions

References

Custom policy-based cost estimation released for optimization of tiered storage configuration

The optimization of tiered storage configuration supports the custom policy-based cost estimation feature. This allows users to estimate changes in storage costs based on custom policies, in addition to the default policy, to assist in decision-making for configuring tiered storage policy attributes.

2025-03-24

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Zhangjiakou), China (Ulanqab), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), and Germany (Frankfurt)

Custom Policy

Multi-zone storage mode released

Multi-zone storage mode has been launched for MaxCompute, and the original default storage mode has been renamed to single-zone storage. We recommend that you use multi-zone storage mode for data related to enterprise production workloads, which can withstand zone-level failures. In the event of a zone-level failure, multi-zone storage ensures uninterrupted data read and write services while preventing data loss.

2025-03-24

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China East 2 Finance, China (Hong Kong), Singapore, and Indonesia (Jakarta)

Zone-disaster recovery

Clustering optimization recommendation released

The MaxCompute clustering optimization recommendation feature analyzes the recent read and write characteristics of a table and generates corresponding clustering suggestions to help improve job performance and reduce CU consumption. You can decide whether to apply the system-recommended clustering suggestions based on the estimated benefits and recommendation details. Additionally, you can view the daily benefits after applying the suggestions.

2025-03-17

China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), and China (Chengdu) 

Clustering optimization recommendations

February

Feature

Description

Release date

Regions

References

Automatic Materialized View (AutoMV) released

Materialized views can be automatically created based on your job query habits and performance. This improves computing efficiency and reduces repeated computing. This feature is suitable for scenarios where multiple repeated queries are performed on a MaxCompute table and different users are unaware that others are also using the same computational logic.

2025-02-20

Public cloud regions in the Chinese mainland

Automatic materialized view (AutoMV)

Public preview of MaxCompute Query Accelerator (MaxQA)

Control links, cache capabilities, and SQL computing engines are optimized on the basis of an exclusive resource pool for query acceleration. This helps reduce the query response time. This feature is suitable for business scenarios that require low latency and high stability, such as BI, interactive analytics, and near real-time data warehousing.

2025-02-17

All regions

User guide for MaxCompute Query Accelerator (MaxQA)

MaxCompute SQL upgraded to enhance engine capabilities

MaxCompute SQL V50 is launched.

  • Enhanced data types and data construction methods: A scale that is greater than 18 is supported by the DECIMAL data type. The STRUCT expression syntax is supported. This is a new method to construct STRUCT data.

  • Enhanced features: The FIND_IN_SET function supports various types of delimiters, and the query rewrite feature of materialized views supports more operators.

  • Performance improvement: MATERIALIZED CTE is used to improve the performance of CTE statements. Various performance optimizations are performed for shuffle scenarios.

2025-02-10

All regions

Data warehouse engine

Dynamic data masking released

The dynamic data masking feature is provided to support multiple masking policies, such as masking, hashing, character replacement, numeric value rounding, and date rounding. This feature effectively protects sensitive data such as personally identifiable information (PII) in scenarios such as business development and testing, data sharing, and O&M.

2025-02-10

China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Ulanqab), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Malaysia (Kuala Lumpur), Indonesia (Jakarta), and Germany (Frankfurt)

Dynamic data masking

Row-level permission feature updated

The row-level permission feature is updated to support the result cache, materialized view, and local mode scenarios. This update, together with other optimized features, improves the usability and performance of the row-level permission feature.

2025-02-10

All regions

Row-level access control

Lakehouse and external table capabilities enhanced

  • When you create an external table to parse data in the PARQUET format, implicit conversions are supported for specific data types, such as TINYINT, SMALLINT, and DATETIME.

  • The MAX_PT function can be used to query the latest partition that contains data in an OSS external table.

2025-02-10

All regions

Lakehouse and the external table feature

January

Feature

Description

Release date

Regions

References

Overview page released in the console

An overview page is added to the MaxCompute console to provide information such as a quick start guide, a list of resource instances, followed projects and quotas, alert and risk warnings, and product updates. This helps you quickly get started with O&M and management.

2025-01-20

All regions

2024

November

Feature

Description

Release date

Region

References

Adaptation of LogView 2.0 to MaxFrame

LogView 2.0 allows you to view the execution records and running durations of all directed acyclic graphs (DAGs) submitted in MaxFrame sessions.

2024-11-29

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Ulanqab), and China (Hong Kong)

Use LogView V2.0 to view MaxFrame jobs

Optimization of the materialized view recommendation and management feature

Estimated impact and actual benefit metrics are added to help you determine whether to adopt recommendations and manage existing materialized views.

2024-11-26

All regions

Recommendations and management of materialized views

Release of intelligent job diagnostics

The intelligent job diagnostics feature can be used to perform intelligent diagnostics on SQL jobs based on the execution details of jobs. This helps you identify issues such as job errors or abnormal job runtime in a timely manner, and provides exception analysis results and solutions.

2024-11-18

All regions

Intelligent diagnostics for jobs

Automatic packaging service of MaxFrame

MaxFrame allows you to declare required external dependencies during job development. This simplifies the management of third-party packages in Python development.

2024-11-01

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Ulanqab), and China (Hong Kong)

Automatic packaging service

October

Feature

Description

Release date

Region

References

Purchase of multiple level-1 subscription computing quotas in the same region

MaxCompute allows you to purchase multiple level-1 subscription computing quotas for the same Alibaba Cloud account in the same region. Each level-1 subscription computing quota has independent resources.

2024-10-14

All regions

Subscription

September

Feature

Description

Release date

Region

References

Release of storage cost optimization

The storage cost optimization feature determines whether the storage tier of a table or partition can be changed from standard storage to infrequent access (IA) or long-term storage based on the default optimization policy and the last access time of the table or partition. The default optimization policy recommends that you change the standard storage tier of a table or partition that is not accessed in the previous 30 days to the IA storage tier and change the standard storage tier of a table or partition that is not accessed in the previous 180 days to the long-term storage tier. This feature also estimates the storage cost savings (catalog price) after the storage tier is changed. This helps you configure tiered storage policies.

2024-09-27

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Zhangjiakou), and China (Ulanqab)

Optimization of storage tiers configuration

Optimized UI for quota management

The UI for quota management of subscription computing resources in the MaxCompute console is optimized. The UI provides two tabs for interactions based on the original logic. This helps improve the experience of quota management.

  • Basic Configurations: If time-based scaling is not required for quotas, you can add and manage level-2 quotas on this tab.

  • Scaling Configuration: If time-based scaling is required for quotas, you can configure the type of the scalable CUs for quotas on this tab after you add level-2 quotas and configure other basic parameters of the level-2 quotas on the Basic Configurations tab. A global view is added to the Scaling Configuration tab. This allows you to view CU configurations of all quotas during all periods.

2024-09-11

All regions

Manage quotas for computing resources in the MaxCompute console

August

Feature

Description

Release date

Region

References

Release of storage resource observation

You can view the total storage usage and the storage usage percentages of different storage types in the selected region. You can also view the storage usage trends of different storage types and the detailed table or partition storage information based on the project and the time range that you select. This way, you can determine whether the storage usage unexpectedly spikes. If a spike occurs, you can optimize the top N projects, tables, or partitions that consume the most storage resources at the earliest opportunity.

2024-08-16

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Zhangjiakou), and China (Ulanqab)

Use resource observation

Improved job performance observation

You can view the trend of data amounts that are scanned by jobs per CU-hour, the top N jobs that consume the most resources, and the top N jobs that consume the most time. You can use metrics to quickly locate jobs that consume a large amount of resources or require a long period of time, and manage the jobs in a timely manner. You can also identify the time periods in which the job performance is poor, locate the causes based on the job O&M feature, and troubleshoot related issues in a timely manner.

2024-08-13

All regions

Use resource observation

July

Feature

Description

Release date

Region

References

Job performance observation

This feature allows you to use relevant metrics to check whether the number of jobs or the resource usage unexpectedly surges. If surges occur, you can resolve these issues at the earliest opportunity. This feature also allows you to compare running durations of jobs during different periods and determine whether slow-running jobs cause overall job performance to deteriorate.

2024-07-31

All regions

Use resource observation

Availability of the Start Time, Wait Time, and Execution Duration parameters on the Jobs page and Job Summary tab

Based on these parameters, you can quickly determine whether a job slowly runs because a long period of time is spent waiting for resource allocation. You can further determine whether the slow running of the job is caused by resource insufficiency or resource competition among multiple jobs based on the parameters and job analysis results. Then, you can optimize job execution, adjust job priorities, or manage computing resources based on your business scenarios.

2024-07-04

All regions

Manage jobs

Availability of computing cost optimization in four regions

You can use the computing cost optimization feature to generate a resource configuration optimization plan for the computing resources in a level-1 subscription quota based on the actual number of job requests and expected resource configurations. You can also use this feature to generate a resource configuration optimization plan for a project that changes the pay-as-you-go computing resources to subscription computing resources and view the estimated optimization effect. This helps you further reduce computing costs and improve resource utilization.

2024-07-01

China (Hong Kong), Singapore, Indonesia (Jakarta), and Germany (Frankfurt)

Optimization of computing resource configuration

June

Feature

Description

Release date

Region

References

Row-level access control

MaxCompute supports row-level access control to help manage the permissions of users or roles to access specific data in MaxCompute tables. You can define the rules for matching users and the data that is allowed to access in a MaxCompute table. This way, specific users or roles can view only the data that they have permissions to access. This improves data security and compliance.

2024-06-28

All regions

Row-level access control

May

Feature

Description

Release date

Region

References

Release of the job performance observation feature

For computing jobs, you can use the metrics related to the overall job runtime to check whether the number of jobs or the resource usage unexpectedly surges and check whether job performance meets your expectations. The metrics include the number of jobs, CU usage, and job running durations.

2024-05-08

All regions

Job performance

April

Feature

Description

Release date

Region

References

Row filtering based on window functions

Window functions allow you to filter out rows that are not used for calculation.

2024-04-10

All regions

filter_clause

Binary constants

Binary constants are supported to provide an easy method to use data of the BINARY type. You can use X'num' to represent binary constants.

2024-04-10

All regions

MaxCompute data type system version 2.0

FROM_CHARSET

The FROM_CHARSET function is added to convert binary data that is in a specified encoding format into a UTF-8 encoded string for subsequent calculation.

2024-04-10

All regions

FROM_CHARSET

TIMESTAMP_NTZ data type

The TIMESTAMP_NTZ data type is added. Dates and time points of the TIMESTAMP_NTZ type do not contain time zones and are not affected by the current system time zone.

2024-04-10

All regions

Introduction to the TIMESTAMP_NTZ data type in MaxCompute

March

Feature

Description

Release date

Region

References

Activation of MaxCompute in the China (Ulanqab) region

MaxCompute can be activated in the China (Ulanqab) region to provide an enterprise-level serverless intelligent data warehousing service. You can activate the service in the MaxCompute console based on your business requirements.

2024-03-26

China (Ulanqab)

-

Migration of ActionTrail event data to MaxCompute

The ActionTrail service allows you to trace and record the event data of your Alibaba Cloud account within the previous 90 days. If you want to analyze event data that was generated a longer period of time ago, you can create a trail by using ActionTrail to deliver the event data to MaxCompute. Then, you can query and analyze the event data.

2024-03-14

All regions

Migrate ActionTrail event data to MaxCompute

Release of SQL Analysis

The SQL Analysis feature of MaxCompute is supported. You can use this feature to perform the following operations: Edit and execute SQL statements, and analyze the execution results in charts. View the metadata of tables, resources, and user-defined functions (UDFs) in all projects, including the metadata of views provided by the built-in tenant-level Information Schema service. Open an SQL file of the built-in public dataset demo and run the code to use and test MaxCompute based on the public datasets.

2024-03-14

All regions

SQL Analysis

January

Feature

Description

Release date

Region

References

Release of computing cost optimization

The computing cost optimization feature is released to replace the computing resource optimization feature of MaxCompute. The computing cost optimization feature is developed based on the computing resource optimization feature. Compared with the computing resource optimization feature, the computing cost optimization feature can generate a resource configuration optimization plan for changing the pay-as-you-go computing resources of a project to subscription computing resources and allows you to view the estimated optimization effect. This helps you further optimize computing costs and improve resource utilization.

2024-01-25

All regions

Optimization of computing resource configuration

Change of the metric aggregation algorithm on the Resource Observation page in the MaxCompute console

The Resource Observation page in the MaxCompute console is provided to improve user experience on the chart display. A maximum of 60 time points can be displayed for each metric in a chart. Therefore, if the selected time range is greater than one hour, the average aggregation algorithm is used and the aggregation data within the time range is displayed in a chart by default. The average aggregation data is the number of minutes within the selected time range divided by 60. You can change the average aggregation algorithm to the maximum value aggregation algorithm based on your business requirements for a more comprehensive analysis of resource consumption.

2024-01-02

All regions

Use resource observation

2023

December

Feature

Description

Release date

Region

References

Release of the Spot Edition

The Spot Edition of MaxCompute provides pay-as-you-go computing resources with lower unit prices. This reduces the cost of using MaxCompute in latency-insensitive scenarios.

2023-12-28

China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Hong Kong), Singapore, UK (London), Japan (Tokyo), Malaysia (Kuala Lumpur), Germany (Frankfurt), Indonesia (Jakarta), US (Virginia), and US (Silicon Valley)

Spot Edition

Editing of public IP addresses in the MaxCompute console

You can add or remove public IP addresses that are available for projects in the MaxCompute console.

2023-12-20

All regions

Network connection process

November

Feature

Description

Release date

Region

References

Configuration for the processing of dirty data in Tablestore external tables

The logic for processing dirty data can be configured when data is read from a Tablestore external table.

2023-11-22

All regions

Create a Tablestore external table

Apache Paimon external tables

An Apache Paimon external table in MaxCompute can be used to read data from Apache Paimon.

2023-11-22

All regions

Apache Paimon external tables

Column-level data encryption

Keysets, key rotation, and data encryption and decryption based on Key Management Service (KMS) keys are supported for column-level encryption.

2023-11-22

All regions

Encryption and decryption functions

Configuration of Super_Administrator role members based on RAM permission control

RAM users that have the UpdateUsersToSuperAdmin permission can be assigned the Super_Administrator role in a project.

2023-11-21

All regions

Project management

September

Feature

Description

Release date

Region

References

MapReduce jobs in SQL runtime execution mode

MapReduce jobs in MaxCompute can run in SQL runtime execution mode.

2023-09-25

All regions

MapReduce jobs in SQL runtime execution mode

Job analytics

The job analytics feature is supported in MaxCompute to help developers identify job resource issues.

2023-09-21

All regions

Manage jobs

August

Feature

Description

Release date

Region

References

Tenant-level Information Schema

Metadata such as project metadata and usage history is provided from the perspective of tenants in Tenant-level Information Schema. Tenant-level Information Schema allows you to obtain all metadata of a type of object of a tenant at a time.

2023-08-08

All regions

Tenant-level Information Schema

June

Feature

Description

Release date

Region

References

JSON data types

JSON data types are supported in MaxCompute. When JSON data is written to a MaxCompute table, MaxCompute automatically performs storage optimization to improve the computing and analysis performance of JSON data.

2023-06-27

All regions

JSON data type

Enhanced capabilities of ZORDER BY for data writing

The capabilities of the ZORDER BY clause in MaxCompute are enhanced. After the enhancement, the ZORDER BY clause supports global sorting when this clause is executed to write data to a table. This reduces the amount of data that is scanned and improves computing performance.

2023-06-27

All regions

Insert data into or overwrite data in a table or a static partition (INSERT INTO and INSERT OVERWRITE)

ALTER TABLE CLEAR COLUMN

The ALTER TABLE CLEAR COLUMN statement can be executed in MaxCompute to clear data from a specific table or columns in specific partitions. This helps save storage space.

2023-06-27

All regions

UPDATE and DELETE

Addition of eight built-in complex type functions

Eight built-in complex type functions are added to MaxCompute. These functions help simplify user operations on arrays and improve development efficiency.

2023-06-27

All regions

May

Feature

Description

Release date

Region

References

PutRow operation for Tablestore external tables

The PutRow operation is supported for Tablestore external tables in MaxCompute.

2023-05-06

All regions

Create a Tablestore external table

Support for the first row specified as a table header

When MaxCompute is used to write data to a CSV file in Object Storage Service (OSS), the first row can be specified as a table header.

2023-05-06

All regions

ORC external tables

New built-in functions

Four mathematical functions, three date functions, and one string function are added to MaxCompute. You can use these built-in functions to simplify SQL code.

2023-05-06

All regions

Scheduled data updates of materialized views

Multiple scheduled data update methods are supported for materialized views. These methods allow the latest data to be stored in materialized views.

2023-05-06

All regions

Materialized view operations

Built-in encryption and decryption functions

Built-in encryption and decryption functions are provided in MaxCompute SQL. You can use these functions to manually encrypt or decrypt columns in tables.

2023-05-06

All regions

Encryption and decryption functions

QUALIFY

The QUALIFY syntax can be used in MaxCompute to filter the results of window functions based on specified filter conditions. This simplifies SQL code in MaxCompute.

2023-05-06

All regions

QUALIFY

TABLESAMPLE

In MaxCompute, TABLESAMPLE can be used to sample table data in different methods.

2023-05-06

All regions

TABLESAMPLE

April

Feature

Description

Release date

Region

References

Schema evolution

The schema evolution feature can be used to add columns of complex data types, delete columns, change the column sequence, and change the data types of columns.

2023-04-25

All regions

Partition operations

Computing acceleration based on materialized views

After you enable this feature, the system automatically provides the recommended SQL scripts for you to create materialized views. This feature helps accelerate the creation of materialized views, reduces repeated computing operations, and saves computing resources.

2023-04-23

All regions

Recommendations and management of materialized views

Job management in the new MaxCompute console

In the new MaxCompute console, you can manage all MaxCompute jobs that are listed and view job snapshots and current job information.

2023-04-06

All regions

Manage jobs

March

Feature

Description

Release date

Region

References

Processing of unstructured data in external volumes

In MaxCompute, you can run Spark jobs and MapReduce jobs to process unstructured data in external volumes based on OSS.

2023-03-30

All regions

Use MaxCompute external volumes to process unstructured data

Storage of unstructured data in external volumes

You can use external volumes provided by MaxCompute to store unstructured data. External volumes in MaxCompute are management objects that are mapped to OSS directories.

2023-03-30

All regions

External volume operations

February

Feature

Description

Release date

Region

References

Display of fine-grained data objects

Information about fine-grained data objects such as tables, views, and functions can be displayed in the output of the SHOW command.

2023-02-25

All regions

SHOW

January

Feature

Description

Release date

Region

References

Direct data reading based on Hologres external tables

If you enable this feature, the number of connections that are established between MaxCompute and a Hologres instance can be reduced and the data reading speed can be accelerated when you read data directly from the Hologres instance.

2023-01-21

All regions

Hologres external tables

2022

December

Feature

Description

Release date

Region

References

WINDOW keyword in SQL statements

The WINDOW keyword is supported. You can use the WINDOW keyword to specify a custom window only once and reuse the custom window.

2022-12-14

All regions

WINDOW keyword

From clause used together with the UPDATE statement

The UPDATE statement can be used together with the from clause to update data.

2022-12-14

All regions

UPDATE and DELETE

New string function and optimization of string functions and aggregate functions

  • The following built-in string functions are optimized: CAST, SPLIT, and RAND.

  • The window functions NUMERIC_HISTOGRAM and PERCENTILE_APPROX are optimized.

  • The MASK_HASH function is added.

2022-12-14

All regions

Overview

November

Feature

Description

Release date

Region

References

Custom administrator roles

MaxCompute provides the following built-in administrator roles for projects: Admin and Super_Administrator. MaxCompute also allows you to create custom administrator roles. To create a custom administrator role, you can perform the following operations in the MaxCompute console of the new version: In the left-side navigation pane of the MaxCompute console, click Projects. On the Projects page, find the desired project and click Manage in the Actions column. On the page that appears, click the Role Permissions tab and click Create Project-level Role. In the Create Role dialog box, select Admin from the Role Type drop-down list. You can grant only specific administrator permissions to custom administrator roles. For example, you can grant permissions to a role to only manage permissions or to only manage IP address whitelists.

2022-11-15

All regions

MaxCompute permissions

October

Feature

Description

Release date

Region

References

Upgrade of the data storage hierarchy from project.table to project.schema.table to allow MaxCompute to connect to data sources that use the three-level data storage hierarchy

A MaxCompute project is a basic organizational unit of MaxCompute and is used for isolation and access control of multiple users. A MaxCompute project contains objects such as tables, resources, and functions. Before the schema feature is provided by MaxCompute, the objects are directly placed in projects. Projects serve as databases or schemas based on the hierarchy of traditional databases. This may cause misunderstanding and inconvenience to users, especially when a large number of tables or objects exist. MaxCompute provides the schema feature. You can use schemas to classify tables, resources, and functions in projects. If the original hierarchy of data storage is project.schema.table and you want to migrate data to MaxCompute, you can use the schema feature of MaxCompute to directly align the hierarchy with the data source hierarchy without the need to reconstruct your business during the migration. The schema feature helps reduce the workload.

2022-10-13

All regions

Schema-related operations

September

Feature

Description

Release date

Region

References

Creation of Hologres external tables in dual-signature authentication mode

The dual-signature authentication mode is an authentication protocol that is developed based on MaxCompute and Hologres. After the user logon information and signature are used on MaxCompute, authentication data is sent to Hologres. Then, Hologres performs authentication by using the same username based on the protocol that is compatible with the underlying layer of MaxCompute. This way, you can directly access Hologres external tables without the need to configure additional authentication information only if you use the same Alibaba Cloud account to access MaxCompute and Hologres.

2022-09-24

All regions

Hologres external tables

Creation of a MaxCompute table with the same schema as a table in external data sources by using CREATE TABLE LIKE

You can use the MaxCompute lakehouse solution to create a MaxCompute table that has the same schema as a table in external data sources such as E-MapReduce (EMR), Hadoop, and Data Lake Formation (DLF). You can use the CREATE TABLE LIKE statement to migrate table schemas from external data sources to MaxCompute. This helps improve data governance and access performance.

2022-09-23

All regions

Use SQL statements to manage an external project

August

Feature

Description

Release date

Region

References

Automatic deletion of expired partitioned tables

If the lifecycle of partition data in a partitioned table expires, MaxCompute automatically reclaims the partition data. When the data of all partitions is reclaimed, MaxCompute can automatically delete the partitioned table. You can configure parameters to enable the automatic table deletion feature.

2022-08-27

All regions

Lifecycle management operations

New aggregate functions

The following aggregate functions are added: BITWISE_AND_AGG, MIN_BY, and MAX_BY. The BITWISE_AND_AGG function is used to return the bitwise AND value of all input values. The MIN_BY function is used to return the value of a column that corresponds to the row in which the minimum value of another column is located. The MAX_BY function is used to return the value of a column that corresponds to the row in which the maximum value of another column is located.

2022-08-27

All regions

Aggregate functions

Schema copy of external tables

When you create an internal table in MaxCompute, you can use the CREATE TABLE LIKE statement to copy the schema of an external table. This feature helps improve the table creation efficiency.

2022-08-27

All regions

Create a table

New function for materialized views

A function is added to MaxCompute. This function allows you to query the status of materialized views. You can use this function to check whether the data of the current materialized view is the same as the data of the original table. You can also use this function to check whether the data of a partition in the current materialized view is the same as the data of the mapped partition in the original table. If the data is the same, True is returned. If the data is different, False is returned.

2022-08-27

All regions

Materialized view operations

Empty partitions generated in materialized views

When you update the data of a partition in a partitioned materialized view, MaxCompute can generate an empty partition in the partitioned materialized view if the calculation result of the materialized view indicates that the partition does not contain data. This ensures that partitions are consecutively generated.

2022-08-27

All regions

Materialized view operations

Job-level quota groups

MaxCompute allows you to specify a quota group at the job level. This helps you use quota groups in a flexible manner. If specific jobs in a project occupy a large number of resources, the overall timeliness for jobs in the project is affected. For example, data refresh jobs occupy a large number of resources but have low timeliness requirements, and specific algorithm jobs occupy a large number of resources and have high timeliness requirements. In this case, you can specify different quota groups for the jobs to isolate resources between the jobs for data computation. This way, you do not need to create another project to migrate the jobs to the new project and associate a quota group with the project to implement resource isolation.

2022-08-23

All regions

Use quota groups for computing resources

July

Feature

Description

Release date

Region

References

Addition of a regular function

The regular function REGEXP_EXTRACT_ALL is added to MaxCompute. This regular function is used to match all substrings that meet the specified requirements from a string that you want to process at a time and return the substrings as an array. This regular function helps improve the data processing efficiency.

2022-07-14

All regions

String functions

Custom prefixes and extensions of exported file names

When you use the Unload function to export data from MaxCompute to OSS, you can configure prefixes and extensions for the exported data files.

2022-07-14

All regions

UNLOAD

Table split size settings

MaxCompute allows you to configure a split size for tables to control the job parallelism. If resources are sufficient but jobs run at a low speed or if jobs take a large amount of time to wait for resource allocation and resources are insufficient, you can adjust the split size to improve the computing efficiency.

2022-07-14

All regions

SELECT syntax

Addition and performance tuning of window functions

The first_value, last_value, and nth_value window functions are added. Performance tuning of all window functions is improved to significantly increase the computing performance of window functions.

2022-07-14

All regions

Window functions

Addition of aggregate functions

The following aggregate functions are added: BITWISE_OR_AGG, MAP_AGG, MULTIMAP_AGG, MAP_UNION, MAP_UNION_SUM, and HISTOGRAM. You can use these aggregate functions to aggregate the input bits or maps to facilitate data analysis and statistics.

2022-07-14

All regions

Aggregate functions

June

Feature

Description

Release date

Region

References

May

Feature

Description

Release date

Region

References

Separate charging for MaxCompute external tables based on the types of external tables

MaxCompute external tables are separately charged based on the types of external tables. You can view the fees that are incurred by OSS external tables and Tablestore external tables in your bills. This way, you can view the fees that are incurred by different data sources for joint computing.

2022-05-17

All regions

View billing details

March

Feature

Description

Release date

Region

References

DISTRIBUTED MAPJOIN

In specific scenarios, DISTRIBUTED MAPJOIN hints can be used to improve the computing performance and reduce the time required for data computation.

2022-03-17

All regions

DISTRIBUTED MAPJOIN

Enhancement of OSS external tables

When MaxCompute writes data to an OSS external table, MaxCompute can automatically create a directory to store the data. When you create an OSS external table, you can specify the cache capacity for reading files.

2022-03-17

All regions

ORC external tables

New parsing method of JSON data

MaxCompute allows you to enclose a key in JSON data whose value contains a period (.) with brackets and single quotation marks (['']) to parse the JSON data.

2022-03-17

All regions

GET_JSON_OBJECT_TUPLE and JSON_TUPLE

Enhancement of the TRIM, LTRIM, and RTRIM functions

MaxCompute allows you to use the TRIM, LTRIM, and RTRIM functions to remove specified characters from the left side, the right side, or both sides of a string.

2022-03-17

All regions

String functions

Enhancement of the query rewrite feature for materialized views

The query rewrite feature can be enabled for a materialized view to help you rewrite a query statement that includes an OUTER JOIN clause, a UNION clause, or a UNION All clause.

2022-03-17

All regions

Materialized view operations

Skipping of headers or footers of TEXTFILE files

The skip.header.line.count and skip.footer.line.count parameters can be used to skip the first and last rows of a CSV file during data processing in MaxCompute. The parameters can also be used if a CSV file is compressed in the .gz, .bz2, or .lzo format.

2022-03-01

All regions

ORC external tables

Compatibility with Apache Spark 3.1

MaxCompute is compatible with Apache Spark 3.1 in addition to the following Apache Spark versions: 1.6, 2.3, and 2.4.

2022-03-01

All regions

Set up a Linux development environment

February

Feature

Description

Release date

Region

References

Data security management of LogView

A custom parameter can be used to specify whether to display the execution results of jobs in LogView of MaxCompute. This enhances data security.

2022-02-25

All regions

Project operations

Table schema changes

The schema of a table in MaxCompute can be changed. You can add fields of complex data types to a table, delete fields from a table, and change the sequence of fields in a table.

2022-02-23

All regions

Partition operations

January

Feature

Description

Release date

Region

References

Display of MaxCompute external project metadata on DataWorks Data Map

The metadata of MaxCompute external projects can be displayed on DataWorks Data Map.

2022-01-10

Germany (Frankfurt) and Singapore

Lakehouse of MaxCompute

2021

December

Feature

Description

Release date

Region

References

Establishment of a connection from MaxCompute to Hadoop Hive in a virtual private cloud (VPC), or from MaxCompute to Data Lake Formation (DLF) and Object Storage Service (OSS)

MaxCompute allows you to create a network connection to a VPC, a connection to a data source, and an external project that is used to implement the lakehouse solution in an end-to-end manner. You do not need to submit a ticket. The time required is reduced from days to minutes. The lakehouse page is added to the MaxCompute console. This page provides entry points for the creation, development, management, and governance of lakehouses.

2021-12-30

All regions

Lakehouse of MaxCompute

Progressive computing supported by MaxCompute

Progressive computing is supported. The intermediate result data is automatically stored in partitions based on a fixed time granularity. In the next cycle, the data that is computed in the last cycle can be reused. This way, less computing resources are consumed, the scheduling period and costs are reduced, and operation efficiency of jobs is improved.

2021-12-03

All regions

Progressive computation

Enhancement of MaxCompute materialized views

Partitions and clusters can be created based on MaxCompute materialized views. If a materialized view does not contain the partition from which you want to query data when you query the materialized view, you can set enable_auto_substitute to true to allow the system to automatically query data from the source table and return the summary data of the source table and materialized view.

2021-12-01

All regions

Materialized view operations

November

Feature

Description

Release date

Region

References

Commercial release of the VPC connection scheme

MaxCompute can access VPCs by using features, such as external tables, user-defined functions (UDFs), and lakehouses. Before MaxCompute can access a VPC, you must establish a network connection between MaxCompute and the destination IP address or service, such as ApsaraDB for HBase, ApsaraDB RDS, or Hadoop clusters. Before the commercial release of the VPC connection scheme, you must fill in an application form. After the commercial release of the VPC connection scheme, you can add or delete a network connection over a VPC on the Network Connection page in the MaxCompute console. You do not need to fill in an application form and wait for approval. This improves development efficiency. Before you configure a network connection, you must understand the permissions related to network connection management.

2021-11-29

  • China (Beijing)

  • China (Shanghai)

  • China (Zhangjiakou)

  • China (Hangzhou)

  • China (Shenzhen)

Network connection process

Clearance of data from the specified partition in a partitioned table

MaxCompute allows you to manually clear the data of one or more partitions that are specified in a partitioned table. Metadata information about the partitioned table or partitions in the partitioned table is not deleted. MaxCompute also allows you to specify filter conditions to filter the partitions whose data needs to be cleared.

2021-11-23

All regions

Clear data from a partition

October

Feature

Description

Release date

Region

References

ApsaraDB for HBase Performance-enhanced Edition and Lindorm supported by MaxCompute external tables

ApsaraDB for HBase Performance-enhanced Edition is a cloud-hosted database that is provided by LindormTable. This edition is fully compatible with open source HBase. Lindorm is a cloud-native multi-mode hyper-converged database that is designed and optimized for scenarios such as Internet of things (IoT), Internet, and Internet of vehicles (IoV). Lindorm supports unified access and processing of a variety of data, such as wide tables, time series, text, objects, streams, and spaces. In various scenarios, you must import data from ApsaraDB for HBase Performance-enhanced Edition or Lindorm to MaxCompute for data processing, data analysis, and federated query. You must also write data from MaxCompute to ApsaraDB for HBase Performance-enhanced Edition or Lindorm. You can create an external table of ApsaraDB for HBase Performance-enhanced Edition or Linorm in MaxCompute to read data from or write data to ApsaraDB for HBase Performance-enhanced Edition or Lindorm.

2021-10-29

All regions

Lindorm external tables

September

Feature

Description

Release date

Region

References

Service availability of MaxCompute in China South 1 Finance

MaxCompute can be activated in the China South 1 Finance region.

2021-09-14

China South 1 Finance

None

Access to Hadoop clusters for which Kerberos authentication is enabled

Hadoop clusters for which Kerberos authentication is enabled can be accessed by MaxCompute. Kerberos authentication is enabled for Hadoop production clusters of most enterprises. This greatly expands the application scope of the integration of lakehouses and Hadoop clusters.

2021-09-01

  • China (Hangzhou)

  • China (Shanghai)

  • China (Beijing)

  • China (Shenzhen)

  • China (Zhangjiakou)

  • Singapore

None

Reading of Hudi data or data in the Delta Lake format from OSS by using the lakehouse solution

The Delta Lake and Apache Hudi storage mechanisms are commonly used in data lake solutions. These storage mechanisms provide stream processing and batch processing capabilities for data lakes. MaxCompute provides a lakehouse solution that supports Delta Lake and Apache Hudi. This solution is developed based on DLF and OSS. You can use MaxCompute to query real-time data and gain instant insight into the changes of business data.

2021-09-01

  • China (Hangzhou)

  • China (Shanghai)

  • China (Beijing)

  • China (Shenzhen)

  • Singapore

Delta Lake or Apache Hudi storage mechanism based on DLF, ApsaraDB RDS or Realtime Compute for Apache Flink, and OSS

August

Feature

Description

Release date

Region

References

Addition of 16 built-in functions to MaxCompute SQL

The following built-in functions are added to MaxCompute SQL:

  • Complex type functions

    FIELD: obtains the values of member variables in STRUCT data.

  • Date functions

    TO_MILLIS: converts a specific date to a UNIX timestamp in milliseconds.

  • String functions

    • ENCODE: encodes a string in a specific encoding format.

    • FIND_IN_SET: returns the position of a specific string among strings that are separated by commas (,).

    • LOCATE: returns the position of a specific string within another string.

    • PARSE_URL_TUPLE: parses a specific URL and returns multiple parts of the URL.

  • Mathematical functions

    CORR: calculates the Pearson correlation coefficient for two columns of data.

  • Other functions

    • HASH: calculates the hash value of the input parameters.

    • COMPRESS: uses the GZIP algorithm to compress input strings.

    • DECOMPRESS: uses the GZIP algorithm to decompress the input parameters of the BINARY type.

    • NULLIF: returns NULL if the values of expr 1 and expr 2 are the same. Otherwise, expr1 is returned.

    • FORMAT_NUMBER: converts a number into a string of the specified format.

    • SHA: calculates the SHA-1 hash value of a string or a binary value and returns a hexadecimal string.

    • SHA1: calculates the SHA-1 hash value of a string or a binary value and returns the value as a hexadecimal string.

    • SHA2: calculates the SHA-2 family hash value of a string or a binary value. SHA-224, SHA-256, SHA-384, and SHA-512 are supported.

    • CRC32: calculates the cyclic redundancy check (CRC) value of a string or binary value.

2021-08-31

All regions

Commercial release of UPDATE, DELETE, and MERGE INTO in MaxCompute SQL

As of August 10, 2021, the public preview of the UPDATE, DELETE, and MERGE INTO statements of MaxCompute SQL ends. After the public preview ends, you are charged when you use these statements. Pay-as-you-go jobs that already use the UPDATE, DELETE, and MERGE INTO statements are charged after the public preview ends. If you did not use the UPDATE, DELETE, or MERGE INTO statement due to concerns about service stability in the public preview phase, you no longer need to worry about stability when you use these statements. MaxCompute provides the same guarantees for the availability and stability of the UPDATE, DELETE, and MERGE INTO statements as the guarantee for MaxCompute SQL.

2021-08-10

  • China (Beijing)

  • China (Shanghai)

  • China (Zhangjiakou)

  • China (Hangzhou)

  • China (Shenzhen)

  • China (Chengdu)

Commercial release of MaxCompute Streaming Tunnel

MaxCompute Streaming Tunnel supports streaming semantic APIs to simplify the development of distributed services. MaxCompute Streaming Tunnel supports the concurrent creation of partitions for distributed services and asynchronous sorting by using the ZORDER BY clause.

2021-08-09

All regions

Overview of streaming tunnel

Materialized views of MaxCompute SQL

MaxCompute SQL supports the materialized view feature. A materialized view is a database object that stores the pre-calculation results of a time-consuming query operation, such as JOIN or AGGREGATE. You can reuse the pre-calculation results when you want to perform the same query. This way, queries are accelerated.

Materialized views are suitable for the following queries:

  • Queries that are in a fixed mode and are frequently executed

  • Queries that involve time-consuming operations, such as JOIN or AGGREGATE

After materialized views are created, MaxCompute can automatically match the optimal materialized view based on the query rewrite capabilities of MaxCompute SQL and read data from the materialized view. This significantly improves query efficiency. You do not need to modify existing queries.

2021-08-06

All regions

Materialized view operations

July

Feature

Description

Release date

Region

References

Pre-sorting of input data of specific aggregate functions in MaxCompute SQL

If you specify WITHIN GROUP (ORDER BY col1[, col2...]) in the aggregate function WM_CONCAT or COLLECT_LIST, or a user-defined aggregate function (UDAF) that is sensitive to the data input order in MaxCompute SQL, the order of the input data of the aggregate function is ensured.

2021-07-30

All regions

Aggregate functions

Syntax of multi-column operations in MaxCompute SQL subqueries

MaxCompute SQL is compatible with the PostgreSQL subquery syntax and provides the IN SUBQUERY/SCALAR SUBQUERY syntax that supports multi-column operations.

  • IN SUBQUERY:

    • Use a simple SELECT statement in which you specify multiple columns for the IN SUBQUERY expression.

    • Use aggregate functions for the IN SUBQUERY expression.

    • Use constants for the IN SUBQUERY expression.

  • SCALAR SUBQUERY:

    • A SELECT list is a SCALAR SUBQUERY expression in which you specify multiple columns. The expression must be an equality expression.

    • Columns in a SELECT list can be an expression of the BOOLEAN type. Only equivalent comparison is supported.

    • A WHERE clause supports multi-column comparison. Only equivalent comparison is supported.

2021-07-29

All regions

Subqueries

New methods to delete MaxCompute projects

A MaxCompute project can be deleted by using an Alibaba Cloud account or a RAM user that is assigned the Super_Administrator role for the project in the MaxCompute console. You can use one of the following methods to delete a MaxCompute project:

  • Logically Delete and Project Restoration Allowed Within 15 Days: If you use this method to delete a project, the project becomes unavailable after it is deleted. To restore the project, find the project and click Restore in the Actions column on the Project management tab within 15 days. After 15 days, the project is permanently deleted and cannot be restored.

  • Immediately Delete and Project Restoration Not Allowed: If you use this method to delete a project, the project is permanently deleted and cannot be restored. If you use this method, you can create a project with the same name immediately after the deletion.

2021-07-29

  • China (Beijing)

  • China (Shanghai)

  • China (Zhangjiakou)

  • China (Hangzhou)

  • China (Shenzhen)

  • China (Chengdu)

Delete a MaxCompute project

April

Feature

Description

Release date

Region

References

Dynamic filtering

Join operations are common in distributed systems. A join operation is both time- and resource-consuming in scenarios in which a large amount of data is processed. For join scenarios, MaxCompute provides the dynamic filtering feature. This feature allows MaxCompute to dynamically generate filters based on runtime by using the equi-join attribute of join operations. This way, MaxCompute can filter data before shuffle operations or join operations. This accelerates queries. The feature is suitable for the join between a dimension table and a fact table. You can configure a parameter to forcibly enable this feature, intelligently enable this feature, or manually use this feature based on displayed messages. You can also perform dynamic partition pruning to filter partitions before you use dynamic filtering.

2021-04-07

  • China (Beijing)

  • China (Shanghai)

  • China (Zhangjiakou)

  • China (Hangzhou)

  • China (Shenzhen)

  • China (Chengdu)

Dynamic filtering

March

Feature

Description

Release date

Region

References

Monitoring and alerting on job timeout for all jobs or all SQL jobs of a MaxCompute project

MaxCompute allows you to configure a threshold-triggered alert rule by using CloudMonitor to monitor the job runtime. If the runtime of a job exceeds the threshold specified in the rule, MaxCompute sends an alert notification to the alert contact. This mechanism helps you identify abnormal jobs at the earliest opportunity and improves operations and maintenance (O&M) efficiency.

2021-03-16

  • China (Beijing)

  • China (Shanghai)

  • China (Zhangjiakou)

  • China (Hangzhou)

  • China (Shenzhen)

  • China (Chengdu)

Monitoring and alerting on job timeout

FROM_JSON and TO_JSON functions supported by MaxCompute SQL

The FROM_JSON and TO_JSON functions are added. You can use the FROM_JSON function to convert data in the JSON format into data of a data type that is supported by MaxCompute. You can also use this function to extract information from data in the JSON format and return data of the ARRAY, MAP, or STRUCT type based on the jsonStr and schema information. You can use the TO_JSON function to convert data in the ARRAY, MAP, or STRUCT format into data in the JSON format.

2021-03-16

All regions

String functions

UPDATE, DELETE, and MERGE INTO statements supported by MaxCompute SQL (public preview)

UPDATE and DELETE statements are supported by MaxCompute to perform row-level operations on data in tables or partitions. Before the UPDATE and DELETE statements are supported, you must read full data, update the data based on join or other operations, and then execute the INSERT OVERWRITE statement to write back the updated data even if you need to update only a small amount of data in a table or partition. The UPDATE and DELETE statements can help you significantly reduce the amount of data that needs to be read or written.

If you want to perform INSERT, UPDATE, and DELETE operations on a table, you can execute the MERGE INTO statement, which scans the full table once and encapsulates all the operations. This improves execution efficiency. In addition, the MERGE INTO statement is atomic. If any internal logic processing fails, the overall job fails to run. This helps avoid the issue that some logic of the same batch of operations cannot be rolled back or redone. Only transactional tables support the UPDATE, DELETE, and MERGE INTO statements. A big data system ensures the atomicity, consistency, isolation, and durability (ACID) semantics for jobs. If multiple jobs run on the same table at the same time, job conflicts may occur. The UPDATE, DELETE, and MERGE INTO statements are in public preview. You are not charged for computing resources when you use these statements. The availability and stability are not guaranteed for the jobs and data that use these statements for production during public preview. We recommend that you back up important data.

2021-03-16

All regions

February

Feature

Description

Release date

Region

References

Commercial release of the lakehouse solution of MaxCompute

The lakehouse solution of MaxCompute is implemented by using a MaxCompute data warehouse and a data lake such as OSS or Hadoop Distributed File System (HDFS). You can build a lakehouse by using one of the following methods:

  • Build a lakehouse by using MaxCompute, DLF, and OSS: The lakehouse solution is implemented by MaxCompute and OSS. If you use this method, you must use DLF. All the metadata (schemas) of the data lake is managed in DLF. MaxCompute can use the metadata management capability of DLF to efficiently process OSS semi-structured data. The OSS semi-structured data includes data in the AVRO, CSV, JSON, Parquet, and ORC formats.

  • Build a lakehouse by using MaxCompute and HDFS: The lakehouse solution is implemented by MaxCompute and HDFS. You can use a Hadoop cluster that is deployed in a data center, on virtual machines (VMs) in the cloud, or in Alibaba Cloud E-MapReduce (EMR). You must fill in an application form to apply for the use of the lakehouse solution of MaxCompute. After you submit the application, MaxCompute technical support personnel contact you and assist you to complete subsequent operations.

2021-02-26

  • China (Beijing)

  • China (Shanghai)

  • China (Hangzhou)

Lakehouse of MaxCompute

ApsaraDB for HBase external tables (public preview)

MaxCompute can access ApsaraDB for HBase over a VPC. After you complete network connection and authorization, you can create an ApsaraDB for HBase external table and read data from and write data to the tables in an ApsaraDB for HBase database by calling the HBaseStorageHandler interface provided by Hive. You can use the ApsaraDB for HBase external table to synchronize data from an ApsaraDB for HBase database to MaxCompute for subsequent extract, transform, and load (ETL) operations. You can also associate the ApsaraDB for HBase external table with a MaxCompute internal table for federated computing, or export MaxCompute data to a table in the ApsaraDB for HBase database. Reading and writing data of ApsaraDB for HBase external tables is in the public preview stage. Data computing is free of charge but the SLA is not guaranteed.

2021-02-08

  • China (Beijing)

  • China (Shanghai)

  • China (Zhangjiakou)

  • China (Hangzhou)

ApsaraDB for HBase external tables (Standard Edition or Performance-enhanced Edition)

Disabling of header display in the MaxCompute CLI to facilitate shell calls

In a shell window or Windows Command Prompt, you may need to use the value that is dynamically returned by the odpscmd -e SQL statement. The shell variable obtains the dynamic return value and then runs subsequent jobs. In this scenario, only field values are required. Other information, such as the runtime information and headers, cannot be returned. You can configure set odps.sql.select.output.format={needHeader:false,fieldDelim:""}; to disable the header display and allow the stdout part of the calculation result to be displayed in the destination handle.

2021-02-08

All regions

MaxCompute client (odpscmd)

Enhancement of data writing to OSS by using OSS external tables

MaxCompute can use the multipart upload feature of OSS to improve data writing efficiency when you perform an INSERT operation to write data to an OSS external table. The data that MaxCompute writes to the OSS external table is stored in the .odps folder of the LOCATION directory, and a .meta file is maintained to ensure the consistency of MaxCompute data. Only MaxCompute can correctly process the data in the .odps folder. If another engine reads the data, an error may be reported. The odps.sql.unstructured.oss.commit.mode parameter is added for MaxCompute. If you set this parameter to true, MaxCompute uses the multipart upload feature to ensure data consistency in a two-phase commit manner. In addition, the .odps folder and the .meta file do not exist. This allows other data processing engines to read data. The default value of this parameter is false.

2021-02-08

All regions

Write data to OSS

Custom Serde classes supported by MaxCompute external tables in the Hive-compatible data type edition

MaxCompute is compatible with the Hive Serde interface to process data in open source formats. MaxCompute also provides default built-in Serde classes to process common data formats. If you want to specify custom ROW FORMAT SERDE to support your data format, you must specify required resources, add a JAR package, and define the method of using jar when you create an external table. This way, you can use custom Serde classes to make external tables compatible with special data formats.

2021-02-08

All regions

Open source data formats supported by OSS external tables

Addition or enhancement of MaxCompute built-in functions

Some built-in functions are added or enhanced.

  • The capabilities of date functions DATE, DATEDIFF, DATEPART, and DATETRUNC are enhanced to support data of the DATE and TIMESTAMP types.

  • The string function PARSE_URL is added to parse URLs. The BASE64 and UNBASE64 functions are added to convert a string between the binary and Base64 formats.

  • Filter expressions are supported for aggregate functions. You can specify filter conditions for an aggregate function in a SELECT clause. This allows you to control the data range of an aggregate function in the same aggregate statement. The COUNT_IF function is added to count the number of data records that meet the IF condition.

  • The STACK function is added to other functions. This function is used to separate the specified data into n rows. This function is compatible with the function usage of Hive and Spark SQL. The GET_USER_ID function is added to other functions. This function is used to obtain the ID of the current Alibaba Cloud account. The ARRAY_INTERSECT function is added to calculate the intersection of two arrays. The isAsc parameter is added to the SORT_ARRAY function. This parameter is used to specify whether to sort the given array in ascending or descending order. The default sorting order is ascending.

2021-02-08

All regions

Introduction of the optimizers Freeride and Analyze to collect metadata

Two optimizers Freeride and Analyze are introduced to collect metadata from tables:

  • Analyze: collects metadata in asynchronous mode. You must run related commands to proactively collect metadata from a specified table in asynchronous mode. Fees are incurred when table data is scanned.

  • Freeride: collects metadata in synchronous mode. You must configure parameters before a CREATE TABLE ... AS ... or INSERT statement and configure the collection plan to enable the Freeride feature. This way, column statistics are automatically collected during data generation. This method is automated but may prolong the query latency.

2021-02-08

All regions

Optimizer

ApsaraDB RDS external tables (public preview)

MaxCompute can access your ApsaraDB RDS instance over a VPC. After you complete a network connection and authorization, you can create an external table of ApsaraDB RDS and read and write data of tables in ApsaraDB RDS. You can use an external table of ApsaraDB RDS to synchronize data such as dimension data and service data from ApsaraDB RDS to MaxCompute for subsequent ETL operations. You can also associate the external table of ApsaraDB RDS with a MaxCompute internal table for federated computing, or export MaxCompute data to ApsaraDB RDS databases. Reading and writing data of ApsaraDB RDS external tables is in the public preview stage. Data computing is free of charge but the SLA is not guaranteed.

2021-02-08

All regions

RDS MySQL external tables

Data reading from or writing to MC-Hologres external tables by using the Java Database Connectivity (JDBC) driver

You can use MC-Hologres external tables to access data of MC-Hologres data sources by using the JDBC driver. You can create an MC-Hologres external table in MaxCompute to query the data of MC-Hologres data sources by using the JDBC driver provided by PostgreSQL. This method prevents redundant data storage and allows you to obtain query results at a fast speed without the need to import or export data. Reading and writing data of MC-Hologres external tables is in the public preview stage. Data computing is free of charge but the Service Level Agreement (SLA) is not guaranteed.

2021-02-08

All regions

Hologres external tables

Data export from MaxCompute to OSS by using UNLOAD

MaxCompute allows you to export data to OSS by using the UNLOAD command. This way, you can use OSS to store structured data and use other computing engines in OSS to process and analyze the data that is exported from MaxCompute.

2021-02-08

All regions

UNLOAD

Commercial release of MaxCompute SQLML

MaxCompute SQLML allows data engineers, analysts, and data scientists to use SQL to create, train, and apply machine learning models in MaxCompute. This enables SQL practitioners to use existing SQL tools and skills to apply machine learning capabilities without the need for data migration. This achieves inclusiveness of machine learning.

2021-02-01

All regions

January

Feature

Description

Release date

Region

References

Adjustment of unit prices of MaxCompute storage services

The tiered pricing of MaxCompute storage services is changed to fixed pricing from January 10, 2021. The new pricing reduces MaxCompute storage costs for users that use small- and medium-sized data. Before the adjustment, tiered pricing is used for MaxCompute storage services. For data of less than or equal to 10 TB, the unit price is CNY 0.0072 per GB per day. For data of greater than 10 TB and less than or equal to 100 TB, the unit price is CNY 0.006 per GB per day. For data of greater than 100 TB, the unit price is CNY 0.004 per GB per day. The preceding unit prices are uniformly changed to CNY 0.12 per GB per month, which is CNY 0.004 per GB per day.

2021-01-10

All regions

Storage pricing

2020

December

Feature

Description

Release date

Region

References

Information Schema service automatically provided for new projects

By default, the Information Schema service is provided for new projects to allow users to view and use project data. Administrators do not need to manually install this service for new projects.

2020-12-21

All regions

Project-level Information Schema (to be discontinued)

Private preview of Information_Schema.Tasks

The Information_Schema.Tasks view is in private preview. This view provides detailed information about the jobs that are running, including the job list and the CPU utilization, memory usage, resource usage, and running duration of each job. This view helps users quickly locate jobs that run slowly and jobs that consume a large number of resources to optimize jobs or plan resource capacity.

2020-12-08

None

Metadata views

Improvement of data analysis capabilities based on MCQA supported by the MaxCompute query editor

MaxCompute Query Acceleration (MCQA) is supported for data queries of MaxCompute. Data queries of MaxCompute are originally performed in offline mode. After MCQA is enabled, the time that is used to run small- and medium-sized query jobs is reduced from minutes to seconds. If you execute an SQL statement to query data by using the MaxCompute query editor, MCQA is preferentially used. If the MCQA requirements are met, results are returned within seconds. If the MCQA requirements are not met, the MCQA job is rolled back to the offline query mode. This ensures that the query job runs as expected. Data analysts can perform data queries, secondary analysis of results, and data sharing based on the powerful analysis features of Web Excel provided by the MaxCompute query editor. The MCQA feature supports pay-as-you-go resources. You cannot enable this feature for a project that uses subscription resources.

2020-12-05

All regions

DataWorks

November

Feature

Description

Release date

Region

References

Data import by using MaxCompute sink connectors provided by ApsaraMQ for Kafka

MaxCompute is integrated with ApsaraMQ for Kafka. You can use the MaxCompute sink connectors provided by ApsaraMQ for Kafka to continuously import data from specific topics to MaxCompute tables. This process requires no third-party tools or secondary development, which significantly simplifies data import from ApsaraMQ for Kafka to MaxCompute and reduces both development and O&M costs.

2020-11-27

All regions

Import Kafka data to MaxCompute in offline or real-time mode

Service access as a RAM role

You are allowed to access a MaxCompute project by using an Alibaba Cloud account, a RAM user, or a RAM role. The administrator can add a RAM role to a project and grant permissions to the role. This way, you can log on to the project as the RAM role by using Security Token Service (STS) without the need to use an AccessKey pair for identity authentication. The MaxCompute console supports STS-based authentication. After authentication, you can use a RAM role to access a MaxCompute project in the MaxCompute console for data analysis and data development.

2020-11-24

All regions

User authentication

June

Feature

Description

Release date

Region

References

Selection of MaxCompute data type editions for new projects

MaxCompute data type editions can be selected for new projects. When you create a MaxCompute project, you must select the initial data type edition for the project.

2020-06-30

All regions

Data type editions

April

Feature

Description

Release date

Region

References

Policy-based access control by using the GRANT statement

The GRANT statement can be executed to perform policy-based access control. This facilitates the authorization operation.

2020-04-23

All regions

Policy-based access control

March

Feature

Description

Release date

Region

References

Addition of five built-in functions

The following built-in functions of MaxCompute are added: TABLE_EXISTS(), PARTITION_EXISTS(), MUMERIC_HISTOGRAM(), PERCENTILE_APPROX(), and FORMAT_NUMBER().

2020-03-17

All regions

February

Feature

Description

Release date

Region

References

Disabling of the lifecycle for a table

The disable lifecycle parameter can be configured for a table to disable the lifecycle for the table.

2020-02-26

All regions

Lifecycle management operations

CLONE TABLE command

The CLONE TABLE command can be used to clone data from one table to another.

2020-02-26

All regions

CLONE TABLE

Extension of GROUPING SETS

The GROUPING SETS clause of MaxCompute is extended. CUBE and ROLLUP can be nested in the GROUPING SETS clause.

2020-02-26

All regions

GROUPING SETS

Acquisition of data from external tables over HTTPS

The set odps.sql.unstructured.data.oss.use.https=true; configuration can be committed together with an SQL statement to obtain data over HTTPS.

2020-02-26

All regions

ORC external tables

MSCK REPAIR TABLE supported by external tables

The MSCK REPAIR TABLE syntax can be executed to add partitions to an Object Storage Service (OSS) external table based on the OSS directory that is specified when you create the OSS external table.

2020-02-26

All regions

ORC external tables

January

Feature

Description

Release date

Region

References

Support for DATE and CHAR data types

The DATE and CHAR data types are supported. If you enable the MaxCompute V2.0 data type edition, you can create a table that contains data of the DATE and CHAR types and data of the DATE and CHAR types can be read.

2020-01-03

All regions

Data type editions

2019

December

Feature

Description

Release date

Region

References

Open source geospatial UDFs

Open source geospatial user-defined functions (UDFs) are supported to analyze spatial data.

2019-12-25

All regions

Geospatial UDFs

November

Feature

Description

Release date

Region

References

Connection to a MaxCompute project by using Tableau

The MaxCompute Java Database Connectivity (JDBC) driver can be used to connect Tableau to a MaxCompute project. This way, MaxCompute data can be displayed on Tableau for analysis.

2019-11-18

All regions

Configure MaxCompute JDBC on Tableau

MaxCompute JDBC driver

The MaxCompute JDBC driver is supported to provide a standard JDBC API. You can perform distributed computing and queries on large amounts of data in MaxCompute by using the JDBC API.

2019-11-18

All regions

Overview

October

Feature

Description

Release date

Region

References

Addition or dropping of multiple partitions at the same time

The DDL semantics of MaxCompute is upgraded. This way, you can add or drop multiple partitions at the same time. You can manage multiple partitions at the same time to improve the efficiency of data warehouse management.

2019-10-28

All regions

Service notices

New operators

The IS [NOT] DISTINCT FROM operator and the string concatenation operator || are supported.

2019-10-28

All regions

Service notices

NATURAL JOIN

The NATURAL JOIN operation is supported to improve the join capability of the MaxCompute SQL engine.

2019-10-28

All regions

JOIN

Configuration of the default values of columns in a table

The DEFAULT VALUE clause is provided to specify a default value for a column when you create a table.

2019-10-28

All regions

Service notices

OFFSET used with ORDER BY LIMIT

An OFFSET clause can be used together with an ORDER BY LIMIT clause to improve the capabilities of the MaxCompute SQL engine.

2019-10-28

All regions

SELECT syntax

Parameters odps.sql.orderby.position.alias and odps.sql.groupby.position.alias

The odps.sql.orderby.position.alias and odps.sql.groupby.position.alias parameters are supported. If you add the set hive.orderby.position.alias=true; command before a SELECT statement, integer constants in the ORDER BY clause are processed as column numbers in the SELECT statement. If you add the set hive.groupby.position.alias=true; command before a SELECT statement, integer constants in the GROUP BY clause are processed as column numbers in the SELECT statement.

2019-10-28

All regions

SELECT syntax

Built-in functions JSON_TUPLE() and EXTRACT()

The built-in functions JSON_TUPLE() and EXTRACT() are supported to improve the SQL processing capabilities.

2019-10-28

All regions

Configuration of the priorities of JOIN operations by using parentheses ()

Parentheses () can be used to specify the priorities of JOIN operations.

2019-10-28

All regions

JOIN

Partition merging

Partitions can be merged. If a large number of partitions exists, you can merge partitions to archive data.

2019-10-28

All regions

Partition operations

September

Feature

Description

Release date

Region

References

Metadata service

The Information Schema service of MaxCompute is supported to provide information such as project metadata and historical data.

2019-09-06

All regions

Project-level Information Schema

Storage price reduction

The storage prices of MaxCompute are reduced from September 1, 2019. The five tiered prices are changed to three tiered prices, and the tiered unit prices are reduced.

2019-09-01

All regions

None

July

Feature

Description

Release date

Region

References

Spark on MaxCompute

Spark on MaxCompute is a computing service that is provided by MaxCompute and compatible with open source Spark. This service provides a Spark computing framework based on unified computing resource and dataset permission systems. This service allows you to use your preferred development method to submit and run Spark jobs. Spark on MaxCompute can fulfill diverse data processing and analytics requirements.

2019-07-23

All regions

Overview

Availability of MaxCompute in the China (Chengdu) region

MaxCompute can be activated in the China (Chengdu) region.

2019-07-04

China (Chengdu)

None

June

Feature

Description

Release date

Region

References

Acquisition of ExecutionContext objects by using UDTs

User-defined types (UDTs) can be used to obtain ExecutionContext objects to access the current ExecutionContext class and resources.

2019-06-25

All regions

Overview

Function overloading supported by UDTs

Generics, class inheritance, and variable-length parameters are supported by UDTs to provide flexible methods to define functions.

2019-06-25

All regions

Overview

Dynamic parameters of UDAFs and UDTFs

Dynamic parameters are supported by user-defined aggregate functions (UDAFs) and user-defined table-valued functions (UDTFs) of MaxCompute to make UDF development more flexible.

2019-06-24

All regions

Dynamic parameters of UDAFs and UDTFs

Partition pruning supported by UDFs

Partition pruning is supported by UDFs to reduce computing costs.

2019-06-24

All regions

None

Parameterized view

The parameterized view feature is supported. This way, you can import tables or variables to create custom views.

2019-06-24

All regions

Parameterized view

Generation of the DDL statement for creating a table

The SHOW CREATE TABLE statement can be executed to generate a DDL statement for creating a table. This way, you can use SQL statements to easily rebuild the table schema.

2019-06-24

All regions

Table operations

Monitoring and alerting of consumption of subscription jobs

The monitoring and alerting feature of MaxCompute is supported. You can configure metrics and alert rules in the CloudMonitor console to monitor the consumption of subscription jobs.

2019-06-21

All regions

Monitoring and alerting