All Products
Search
Document Center

MaxCompute:Updates in 2021

Last Updated:Aug 10, 2023

This topic describes the latest updates to MaxCompute documentation in 2021. These updates provide detailed information about the new features, new syntax, and permission updates in MaxCompute to help you improve the efficiency of project development.

Updates in December 2021

Release date

Feature

Category

Description

References

2021-12-22

Lakehouse solution of MaxCompute

Updated description

The lakehouse solution of MaxCompute can be implemented by using the DataWorks console.

Lakehouse of MaxCompute

Updates in November 2021

Release date

Feature

Category

Description

References

2021-11-29

Policy-based permission management

New practice

Policy-based permission management is supported for users assigned built-in roles.

Policy-based permission management for users assigned built-in roles

2021-11-29

Tenant-level user management

Updated description

Users can be managed in the MaxCompute console.

Role planning

2021-11-15

Conversion of IP addresses to geolocations

New practice

User-defined functions (UDFs) can be used in MaxCompute to convert IPv4 or IPv6 addresses to geolocations.

Use a MaxCompute UDF to convert IPv4 or IPv6 addresses into geolocations

2021-11-10

Python program upload and Python UDF creation

New description

After a Python program is developed and debugged, the Python program must be uploaded to a MaxCompute project as a resource. Then, the Python program can be used to create a MaxCompute UDF.

Upload a Python program and create a MaxCompute UDF

2021-11-10

Visualized function management in MaxCompute Studio

New description

A visual editor is provided by Project Explorer of MaxCompute Studio. The visual editor helps you create UDFs in a MaxCompute project. The editor also provides an entry point that allows you to view the code of MaxCompute UDFs and delete the UDFs.

Manage functions in a visualized manner

2021-11-08

Input parameters of date functions, mathematical functions, window functions, and aggregate functions

Updated description

The description for null values of input parameters of date functions, mathematical functions, window functions, and aggregate functions is added.

2021-11-03

Clearance of partition data in partitioned tables

New description

MaxCompute allows you to clear partition data that meet a specified filter condition. If you want to delete one or more partitions that meet a filter condition at a time, you can use an expression to specify the condition, use the condition to match partitions, and clear the partition data at a time.

Clear data from a partition

2021-11-02

Visualized resource management in MaxCompute Studio

New description

A visual editor is provided by Project Explorer of MaxCompute Studio. The visual editor helps you add the resources that are required to create UDFs in a MaxCompute project. The resources include tables, JAR files, Python scripts, and compressed packages. The editor also provides an entry point that allows you to obtain or remove resources.

Manage resources in a visualized manner

Updates in October 2021

Release date

Feature

Category

Description

References

2021-10-26

Clearance of partition data in partitioned tables

New description

Partition data that meets a specified filter condition can be cleared in MaxCompute. If you want to delete one or more partitions that meet a filter condition at a time, you can use an expression to specify the condition, use the condition to match partitions, and then clear the partition data.

Partition and column operations

2021-10-19

Type conversions of date values

New practice

Data types can be converted among STRING, TIMESTAMP, and DATETIME.

Convert data types among STRING, TIMESTAMP, and DATETIME

Updates in September 2021

Release date

Feature

Category

Description

References

2021-09-30

Content adjustment of MaxCompute UDFs

Updated description

The directory and content of MaxCompute UDFs are reconstructed based on user-defined scalar functions, user-defined table-valued functions (UDTFs), and user-defined aggregate functions (UDAFs). Examples and FAQ are added.

Overview

2021-09-22

Change in the network connection process

Updated description

Some operations in the network connection process are performed by submitting application forms instead of submitting tickets.

Network connection process

2021-09-16

Guide on how to connect business intelligence (BI) tools to MaxCompute

New description

BI tools can be used to visualize analysis results of data from MaxCompute in charts.

2021-09-16

Guide on how to connect database management tools to MaxCompute

New description

Database management tools can be used to manage MaxCompute projects.

2021-09-15

DATE_SUB function

New description

The DATE_SUB function can be used to subtract a number of days from a date. The logic of the DATE_SUB function is opposite to that of the DATE_ADD function.

DATE_SUB

2021-09-15

Spark in Java

Updated description

Examples on how to use Spark in Java are added.

2021-09-08

Concept hierarchy

New description

The concept hierarchy of MaxCompute is added.

Concept hierarchy

Updates in August 2021

Release date

Feature

Category

Description

References

2021-08-26

CORR function

New description

The CORR function is added to mathematical functions. This function calculates the Pearson correlation coefficient for two columns of data.

Mathematical functions

2021-08-20

Syntax updates for aggregate functions

Updated description

The "Syntax" section is added to the "Aggregate functions" topic.

Aggregate functions

2021-08-19

HASH, COMPRESS, DECOMPRESS, NULLIF, FORMAT_NUMBER, SHA, SHA1, SHA2, and CRC32 added to other functions

New description

  • HASH: calculates the hash value of the input parameters.

  • COMPRESS: uses the GZIP algorithm to compress input strings.

  • DECOMPRESS: uses the GZIP algorithm to decompress the input parameters of the BINARY type.

  • NULLIF: returns NULL if the values of expr 1 and expr 2 are the same. Otherwise, expr1 is returned.

  • FORMAT_NUMBER: converts a number into a string of the specified format.

  • SHA: calculates the SHA-1 hash value of a string or a binary value and returns a hexadecimal string.

  • SHA1: calculates the SHA-1 hash value of a string or a binary value and returns the value as a hexadecimal string.

  • SHA2: calculates the SHA-2 family hash value of a string or a binary value. SHA-224, SHA-256, SHA-384, and SHA-512 are supported.

  • CRC32: calculates the cyclic redundancy check (CRC) value of a string or binary value.

Other functions

2021-08-19

TO_MILLIS function

New description

The TO_MILLIS function is added to date functions. This function converts a date into a UNIX timestamp in milliseconds.

Date functions

2021-08-18

Change in the minimum number of MaxCompute CUs

Updated description

The minimum number of MaxCompute compute units (CUs) that you must purchase is changed from 10 CUs to 50 CUs.

Overview

2021-08-11

Commercial use of the UPDATE, DELETE, and MERGE INTO statements

Updated description

As of August 10, 2021, the public preview of the UPDATE, DELETE, and MERGE INTO statements of MaxCompute SQL ends. After the public preview ends, you are charged when you use these statements. Pay-as-you-go jobs that already use the UPDATE, DELETE, and MERGE INTO statements are charged after the public preview ends. If you did not use the UPDATE, DELETE, or MERGE INTO statement due to concerns about service stability in the public preview phase, you no longer need to worry about stability when you use these statements. MaxCompute provides the same guarantees for the availability and stability of the UPDATE, DELETE, and MERGE INTO statements as the guarantee for MaxCompute SQL.

2021-08-10

Functions of complex data types

New feature

MaxCompute can process data of complex data types, such as MAP, ARRAY, and STRUCT, and provides a variety of functions to support diversified data processing.

Complex type functions

2021-08-04

ETL tools

New description

Extract, transform, and load (ETL) tools can be used to manage MaxCompute projects.

2021-08-02

IN/NOT IN SUBQUERY and SCALAR SUBQUERY

Updated description

IN/NOT IN SUBQUERY and SCALAR SUBQUERY support multi-column subqueries, which is a feature in PostgreSQL.

Subqueries

Updates in July 2021

Release date

Feature

Category

Description

References

2021-07-16

Top scheduled SQL tasks that are managed by using a dynamic filter

New feature

If a top scheduled SQL task is found, the ID of the most recent instance on which the task runs is displayed. We recommend that you use a dynamic filter to check and optimize the related job to reduce resource consumption and improve the job running performance.

Use MaxCompute Management

2021-07-15

Limits on packages

Updated description

The limits on packages are added.

Use a package

Updates in June 2021

Release date

Feature

Category

Description

References

2021-06-07

Description of data import and data export by using Data Integration

Updated description

The description of data import and data export by using Data Integration is modified.

Updates in May 2021

Release date

Feature

Category

Description

References

2021-05-31

Descriptions related to preparation items

Updated description

The preparation procedure and related descriptions are modified.

Overview

2021-05-31

Quick start

Updated description

A quick start guide is provided for each tool.

Quick Start

Updates in March 2021

Release date

Feature

Category

Description

References

2021-03-25

MaxCompute error codes

Updated description

New information is added to the MaxCompute error code list.

2021-03-16

NOT NULL attribute for a specific column in the CREATE TABLE syntax

Updated description

The NOT NULL attribute can be configured in the CREATE TABLE syntax to specify that the values in a specific column cannot be NULL.

Table operations

2021-03-16

FROM_JSON and TO_JSON functions

New description

The FROM_JSON and TO_JSON functions are added. You can use the FROM_JSON function to convert data in the JSON format into data of a data type that is supported by MaxCompute. You can also use this function to extract information from data in the JSON format and return data of the ARRAY, MAP, or STRUCT type based on the jsonStr and schema information. You can use the TO_JSON function to convert data in the ARRAY, MAP, or STRUCT format into data in the JSON format.

String functions

2021-03-16

ACID semantics for transactional tables

New feature

New atomicity, consistency, isolation, durability (ACID) semantics for transactional tables are added based on the ACID semantics for concurrent write jobs.

ACID semantics

2021-03-12

A periodic check for scheduled tasks that are continuously run with empty output and scheduled tasks with continuous full table scans

New feature

Top scheduled SQL tasks that are continuously run with empty output and top scheduled SQL tasks with continuous full table scans can be periodically checked in MaxCompute. If these tasks are found, the ID of the most recent instance on which the task runs is displayed. We recommend that you check and optimize the related jobs to reduce resource consumption.

Use MaxCompute Management

2021-03-12

Column-level access control

Updated description

Column-level access control is supported.

MaxCompute permissions

2021-03-12

Enabling and disabling of download control based on property settings

Updated description

A property can be configured to enable or disable download control. To enable or disable download control, you must configure a property at the project level as the project owner or a user who is assigned the Super_Administrator role.

Policy-based access control

2021-03-01

Data lakehouse

New feature

MaxCompute provides the lakehouse solution, which enables you to build a data management platform that combines data lakes and data warehouses. The lakehouse solution integrates the flexibility and diverse ecosystems of data lakes with the enterprise-class deployment of data warehouses.

Lakehouse of MaxCompute

Updates in February 2021

Release date

Feature

Category

Description

References

2021-02-23

Example of how to use a Python 2 UDTF to read resources from MaxCompute

New example

An example of how to use a Python 2 UDTF to read resources from MaxCompute is added.

Python 2 UDFs

2021-02-23

Example of how to use a Python 3 UDTF to read resources from MaxCompute

New example

An example of how to use a Python 3 UDTF to read resources from MaxCompute is added.

Python 3 UDFs

2021-02-23

External files referenced by Spark jobs

New description

The description that Spark jobs reference external files is added.

Set up a Linux development environment

2021-02-20

Example of how to use a UDTF with LATERAL VIEW

New example

An example of how to use a UDTF with LATERAL VIEW is added.

UDTF usage

2021-02-09

Deletion of user permissions

New description

The description for the deletion of user permissions is added.

User planning and management

2021-02-07

Removal of the limit that only constants can be used in VALUES

Updated description

The limit that only constants can be used in VALUES is removed.

VALUES

2021-02-07

Multipart upload of Object Storage Service (OSS) external tables

New description

The multipart upload feature of OSS is supported. This feature allows you to perform the INSERT operation to write data to OSS external tables.

Write data to OSS

2021-02-07

ARRAY_INTERSECT and SORT_ARRAY functions

New description

The ARRAY_INTERSECT function is added to calculate the intersection of two arrays. The isAsc parameter is added to the SORT_ARRAY function. This parameter specifies whether to sort the given arrays in ascending or descending order. The default value of this parameter is True, which indicates that given arrays are sorted in ascending order.

Other functions

2021-02-07

Filter expressions and the COUNT_IF function

New description

Filter expressions are supported. You can specify filter conditions for an aggregate function in a SELECT clause. This way, you can separately control the data range of the aggregate function in an aggregate statement. The COUNT_IF function is added to count the number of data records that meet the IF condition.

Aggregate functions

2021-02-07

Support for the DATE and TIMESTAMP data types in DATEADD, DATE_ADD, DATEDIFF, DATEPART, and DATETRUNC functions

New description

The date functions DATEADD, DATE_ADD, DATEDIFF, DATEPART, and DATETRUNC are enhanced to support the DATE and TIMESTAMP data types.

Date functions

2021-02-07

BASE64, UNBASE64, and PARSE_URL functions

New description

The string function PARSE_URL is added to parse URLs. The BASE64 and UNBASE64 functions are added to implement conversions between binary strings and Base64-encoded strings.

String functions

2021-02-07

Disabling of header display in the MaxCompute command line to facilitate shell calls

New description

In a shell window or in the Windows command line, you may need to use the dynamic return value that is obtained by executing the odpscmd -e SQL statement. The variable of the shell can obtain this dynamic return value and run subsequent jobs in the shell. In this scenario, only field values are required. Other information, such as runtime information and headers, must not be returned. You can run set odps.sql.select.output.format={needHeader:false,fieldDelim:""}; to disable the header display and export the standard output from the calculation result to the destination handle.

MaxCompute client (odpscmd)

2021-02-07

Data export from MaxCompute to OSS by using UNLOAD

New feature

MaxCompute allows you to export data to OSS by using the UNLOAD command. This way, you can use OSS to store structured data and use other computing engines in OSS to process and analyze the data that is exported from MaxCompute.

UNLOAD

2021-02-07

SKEWJOIN HINT

New feature

If two tables that you want to join contain hot key values, a long tail issue may occur. You can remove the hot key value and divide the data into hot data and non-hot data, and then merge them to improve the join efficiency. In this case, SKEWJOIN HINT can be used to automatically or manually extract hot key values from the two tables, separately calculate the join results of the hot key values and non-hot key values, and then join the calculated data. This way, the JOIN operation is accelerated.

SKEWJOIN HINT

2021-02-07

Reading and writing MC-Hologres data by using the JDBC driver

New feature

You can use MaxCompute external tables to access data of MC-Hologres data sources by using the Java Database Connectivity (JDBC) driver. You can create an MC-Hologres external table in MaxCompute to query the data of MC-Hologres data sources by using the JDBC driver that is provided by PostgreSQL. This method prevents redundant data storage and allows you to obtain query results at a fast speed without the need to import or export data. Reading and writing data of MC-Hologres external tables is in the public preview stage. Data computing is free of charge but the Service Level Agreement (SLA) is not guaranteed.

Hologres foreign tables

2021-02-01

Change records of Mars V0.6.3

New description

Change records of Mars V0.6.3 are added.

Version updates

Updates in January 2021

Release date

Feature

Category

Description

References

2021-01-21

Change in the return behavior of the GET_JSON_OBJECT function

Updated description

For new MaxCompute projects, the GET_JSON_OBJECT function retains the original strings instead of escaping JSON reserved characters when the function returns a value.

GET_JSON_OBJECT

2021-01-19

Content adjustment and optimization of the SELECT syntax

Updated description

The content of the SELECT syntax is adjusted, and sample statements are optimized.

SELECT syntax

2021-01-05

Description of data migration

New description

The Data migration topic is added. The content related to data upload and download and the migration tool is moved under this topic.

Data upload and download