This topic describes the latest updates to MaxCompute documentation in 2021. These updates provide detailed information about the new features, new syntax, and permission updates in MaxCompute to help you improve the efficiency of project development.
Updates in December 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-12-24 | External data source management | New description | External data sources can be created to connect to Hadoop clusters. This helps implement the lakehouse solution of MaxCompute. | Manage external data sources |
2021-12-22 | Lakehouse solution of MaxCompute | Updated description | The lakehouse solution of MaxCompute can be implemented by using the DataWorks console. | Lakehouse of MaxCompute |
2021-12-06 | Data visualization | Updated description | A new topic about data visualization on an online operational analytics platform is provided. | Display data on dashboards |
Updates in November 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-11-29 | Policy-based permission management | New practice | Policy-based permission management is supported for users assigned built-in roles. | Policy-based permission management for users assigned built-in roles |
2021-11-29 | Tenant-level user management | Updated description | Users can be managed in the MaxCompute console. | Role planning and management |
2021-11-15 | Conversion of IP addresses to geolocations | New practice | User-defined functions (UDFs) can be used in MaxCompute to convert IPv4 or IPv6 addresses to geolocations. | Use a MaxCompute UDF to convert IPv4 or IPv6 addresses into geolocations |
2021-11-10 | Python program upload and Python UDF creation | New description | After a Python program is developed and debugged, the Python program must be uploaded to a MaxCompute project as a resource. Then, the Python program can be used to create a MaxCompute UDF. | Upload a Python program and create a MaxCompute UDF |
2021-11-10 | Visualized function management in MaxCompute Studio | New description | A visual editor is provided by Project Explorer of MaxCompute Studio. The visual editor helps you create UDFs in a MaxCompute project. The editor also provides an entry point that allows you to view the code of MaxCompute UDFs and delete the UDFs. | Manage functions in a visualized manner |
2021-11-08 | Input parameters of date functions, mathematical functions, window functions, and aggregate functions | Updated description | The description for null values of input parameters of date functions, mathematical functions, window functions, and aggregate functions is added. | |
2021-11-03 | Clearance of partition data in partitioned tables | New description | MaxCompute allows you to clear partition data that meet a specified filter condition. If you want to delete one or more partitions that meet a filter condition at a time, you can use an expression to specify the condition, use the condition to match partitions, and clear the partition data at a time. | Clear data from a partition |
2021-11-02 | Visualized resource management in MaxCompute Studio | New description | A visual editor is provided by Project Explorer of MaxCompute Studio. The visual editor helps you add the resources that are required to create UDFs in a MaxCompute project. The resources include tables, JAR files, Python scripts, and compressed packages. The editor also provides an entry point that allows you to obtain or remove resources. | Manage resources in a visualized manner |
Updates in October 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-10-26 | Clearance of partition data in partitioned tables | New description | Partition data that meets a specified filter condition can be cleared in MaxCompute. If you want to delete one or more partitions that meet a filter condition at a time, you can use an expression to specify the condition, use the condition to match partitions, and then clear the partition data. | Partition and column operations |
2021-10-19 | Type conversions of date values | New practice | Data types can be converted among STRING, TIMESTAMP, and DATETIME. | Convert data types among STRING, TIMESTAMP, and DATETIME |
Updates in September 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-09-30 | Content adjustment of MaxCompute UDFs | Updated description | The directory and content of MaxCompute UDFs are reconstructed based on user-defined scalar functions, user-defined table-valued functions (UDTFs), and user-defined aggregate functions (UDAFs). Examples and FAQ are added. | Overview |
2021-09-22 | Change in the network connection process | Updated description | Some operations in the network connection process are performed by submitting application forms instead of submitting tickets. | Network connection process |
2021-09-16 | Guide on how to connect business intelligence (BI) tools to MaxCompute | New description | BI tools can be used to visualize analysis results of data from MaxCompute in charts. | |
2021-09-16 | Guide on how to connect database management tools to MaxCompute | New description | Database management tools can be used to manage MaxCompute projects. | |
2021-09-15 | DATE_SUB function | New description | The DATE_SUB function can be used to subtract a number of days from a date. The logic of the DATE_SUB function is opposite to that of the DATE_ADD function. | DATE_SUB |
2021-09-15 | Spark in Java | Updated description | Examples on how to use Spark in Java are added. | |
2021-09-08 | Concept hierarchy | New description | The concept hierarchy of MaxCompute is added. | Concept hierarchy |
Updates in August 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-08-26 | CORR function | New description | The CORR function is added to mathematical functions. This function calculates the Pearson correlation coefficient for two columns of data. | Mathematical functions |
2021-08-20 | Syntax updates for aggregate functions | Updated description | The "Syntax" section is added to the "Aggregate functions" topic. | Aggregate functions |
2021-08-19 | HASH, COMPRESS, DECOMPRESS, NULLIF, FORMAT_NUMBER, SHA, SHA1, SHA2, and CRC32 added to other functions | New description |
|
Other functions |
2021-08-19 | TO_MILLIS function | New description | The TO_MILLIS function is added to date functions. This function converts a date into a UNIX timestamp in milliseconds. | Date functions |
2021-08-18 | Change in the minimum number of MaxCompute CUs | Updated description | The minimum number of MaxCompute compute units (CUs) that you must purchase is changed from 10 CUs to 50 CUs. | Overview |
2021-08-11 | Commercial use of the UPDATE, DELETE, and MERGE INTO statements | Updated description | As of August 10, 2021, the public preview of the UPDATE, DELETE, and MERGE INTO statements of MaxCompute SQL ends. After the public preview ends, you are charged when you use these statements. Pay-as-you-go jobs that already use the UPDATE, DELETE, and MERGE INTO statements are charged after the public preview ends. If you did not use the UPDATE, DELETE, or MERGE INTO statement due to concerns about service stability in the public preview phase, you no longer need to worry about stability when you use these statements. MaxCompute provides the same guarantees for the availability and stability of the UPDATE, DELETE, and MERGE INTO statements as the guarantee for MaxCompute SQL. | |
2021-08-10 | Functions of complex data types | New feature | MaxCompute can process data of complex data types, such as MAP, ARRAY, and STRUCT, and provides a variety of functions to support diversified data processing. | Complex type functions |
2021-08-04 | ETL tools | New description | Extract, transform, and load (ETL) tools can be used to manage MaxCompute projects. | |
2021-08-02 | IN/NOT IN SUBQUERY and SCALAR SUBQUERY
|
Updated description | IN/NOT IN SUBQUERY and SCALAR SUBQUERY support multi-column subqueries, which is a feature in PostgreSQL. | Subqueries |
Updates in July 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-07-16 | Top scheduled SQL tasks that are managed by using a dynamic filter | New feature | If a top scheduled SQL task is found, the ID of the most recent instance on which the task runs is displayed. We recommend that you use a dynamic filter to check and optimize the related job to reduce resource consumption and improve the job running performance. | Use MaxCompute Management |
2021-07-15 | Limits on packages | Updated description | The limits on packages are added. | Use a package |
Updates in June 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-06-07 | Description of data import and data export by using Data Integration | Updated description | The description of data import and data export by using Data Integration is modified. |
Updates in May 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-05-31 | Descriptions related to preparation items | Updated description | The preparation procedure and related descriptions are modified. | Overview |
2021-05-31 | Quick start | Updated description | A quick start guide is provided for each tool. | Quick Start |
Updates in March 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-03-25 | MaxCompute error codes | Updated description | New information is added to the MaxCompute error code list. | |
2021-03-16 | NOT NULL attribute for a specific column in the CREATE TABLE syntax | Updated description | The NOT NULL attribute can be configured in the CREATE TABLE syntax to specify that the values in a specific column cannot be NULL. | Table operations |
2021-03-16 | FROM_JSON and TO_JSON functions | New description | The FROM_JSON and TO_JSON functions are added. You can use the FROM_JSON function to convert data in the JSON format into data of a data type that is supported by MaxCompute. You can also use this function to extract information from data in the JSON format and return data of the ARRAY, MAP, or STRUCT type based on the jsonStr and schema information. You can use the TO_JSON function to convert data in the ARRAY, MAP, or STRUCT format into data in the JSON format. | String functions |
2021-03-16 | ACID semantics for transactional tables | New feature | New atomicity, consistency, isolation, durability (ACID) semantics for transactional tables are added based on the ACID semantics for concurrent write jobs. | ACID semantics |
2021-03-12 | A periodic check for scheduled tasks that are continuously run with empty output and scheduled tasks with continuous full table scans | New feature | Top scheduled SQL tasks that are continuously run with empty output and top scheduled SQL tasks with continuous full table scans can be periodically checked in MaxCompute. If these tasks are found, the ID of the most recent instance on which the task runs is displayed. We recommend that you check and optimize the related jobs to reduce resource consumption. | Use MaxCompute Management |
2021-03-12 | Column-level access control | Updated description | Column-level access control is supported. | Permissions |
2021-03-12 | Enabling and disabling of download control based on property settings | Updated description | A property can be configured to enable or disable download control. To enable or disable download control, you must configure a property at the project level as the project owner or a user who is assigned the Super_Administrator role. | Policy-based access control |
2021-03-01 | Data lakehouse | New feature | MaxCompute provides the lakehouse solution, which enables you to build a data management platform that combines data lakes and data warehouses. The lakehouse solution integrates the flexibility and diverse ecosystems of data lakes with the enterprise-class deployment of data warehouses. | Lakehouse of MaxCompute |
Updates in February 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-02-23 | Example of how to use a Python 2 UDTF to read resources from MaxCompute | New example | An example of how to use a Python 2 UDTF to read resources from MaxCompute is added. | Python 2 UDFs |
2021-02-23 | Example of how to use a Python 3 UDTF to read resources from MaxCompute | New example | An example of how to use a Python 3 UDTF to read resources from MaxCompute is added. | Python 3 UDFs |
2021-02-23 | External files referenced by Spark jobs | New description | The description that Spark jobs reference external files is added. | Set up a Spark on MaxCompute development environment |
2021-02-20 | Example of how to use a UDTF with LATERAL VIEW | New example | An example of how to use a UDTF with LATERAL VIEW is added. | UDTF usage |
2021-02-09 | Deletion of user permissions | New description | The description for the deletion of user permissions is added. | User planning and management |
2021-02-07 | Removal of the limit that only constants can be used in VALUES | Updated description | The limit that only constants can be used in VALUES is removed. | VALUES |
2021-02-07 | Multipart upload of Object Storage Service (OSS) external tables | New description | The multipart upload feature of OSS is supported. This feature allows you to perform the INSERT operation to write data to OSS external tables. | Export unstructured data to OSS |
2021-02-07 | ARRAY_INTERSECT and SORT_ARRAY functions
|
New description | The ARRAY_INTERSECT function is added to calculate the intersection of two arrays . The isAsc parameter is added to the SORT_ARRAY function. This parameter specifies whether to sort the given arrays in ascending
or descending order. The default value of this parameter is True, which indicates
that given arrays are sorted in ascending order.
|
Other functions |
2021-02-07 | Filter expressions and the COUNT_IF function
|
New description | Filter expressions are supported. You can specify filter conditions for an aggregate
function in a SELECT clause. This way, you can separately control the data range of the aggregate function
in an aggregate statement. The COUNT_IF function is added to count the number of data records that meet the IF condition.
|
Aggregate functions |
2021-02-07 | Support for the DATE and TIMESTAMP data types in DATEADD , DATE_ADD, DATEDIFF , DATEPART , and DATETRUNC functions
|
New description | The date functions DATEADD , DATE_ADD, DATEDIFF , DATEPART , and DATETRUNC are enhanced to support the DATE and TIMESTAMP data types.
|
Date functions |
2021-02-07 | BASE64 , UNBASE64 , and PARSE_URL functions
|
New description | The string function PARSE_URL is added to parse URLs. The BASE64 and UNBASE64 functions are added to implement conversions between binary strings and Base64-encoded
strings.
|
String functions |
2021-02-07 | Disabling of header display in the MaxCompute command line to facilitate shell calls | New description | In a shell window or in the Windows command line, you may need to use the dynamic
return value that is obtained by executing the odpscmd -e SQL statement. The variable of the shell can obtain this dynamic return value and run
subsequent jobs in the shell. In this scenario, only field values are required. Other
information, such as runtime information and headers, must not be returned. You can
run set odps.sql.select.output.format={needHeader:false,fieldDelim:""}; to disable the header display and export the standard output from the calculation
result to the destination handle.
|
MaxCompute client |
2021-02-07 | Data export from MaxCompute to OSS by using UNLOAD | New feature | MaxCompute allows you to export data to OSS by using the UNLOAD command. This way, you can use OSS to store structured data and use other computing engines in OSS to process and analyze the data that is exported from MaxCompute. | UNLOAD |
2021-02-07 | SKEWJOIN HINT | New feature | If two tables that you want to join contain hot key values, a long tail issue may occur. You can remove the hot key value and divide the data into hot data and non-hot data, and then merge them to improve the join efficiency. In this case, SKEWJOIN HINT can be used to automatically or manually extract hot key values from the two tables, separately calculate the join results of the hot key values and non-hot key values, and then join the calculated data. This way, the JOIN operation is accelerated. | SKEWJOIN HINT |
2021-02-07 | Reading and writing MC-Hologres data by using the JDBC driver | New feature | You can use MaxCompute external tables to access data of MC-Hologres data sources by using the Java Database Connectivity (JDBC) driver. You can create an MC-Hologres external table in MaxCompute to query the data of MC-Hologres data sources by using the JDBC driver that is provided by PostgreSQL. This method prevents redundant data storage and allows you to obtain query results at a fast speed without the need to import or export data. Reading and writing data of MC-Hologres external tables is in the public preview stage. Data computing is free of charge but the Service Level Agreement (SLA) is not guaranteed. | Hologres external tables |
2021-02-01 | Change records of Mars V0.6.3 | New description | Change records of Mars V0.6.3 are added. | Version updates |
Updates in January 2021
Release date | Feature | Category | Description | References |
---|---|---|---|---|
2021-01-21 | Change in the return behavior of the GET_JSON_OBJECT function
|
Updated description | For new MaxCompute projects, the GET_JSON_OBJECT function retains the original strings instead of escaping JSON reserved characters
when the function returns a value.
|
GET_JSON_OBJECT |
2021-01-19 | Content adjustment and optimization of the SELECT syntax | Updated description | The content of the SELECT syntax is adjusted, and sample statements are optimized. | SELECT syntax |
2021-01-05 | Description of data migration | New description | The Data migration topic is added. The content related to data upload and download and the migration tool is moved under this topic. | Data upload and download |