All Products
Search
Document Center

MaxCompute:Service notices

Last Updated:Mar 06, 2024

This topic describes the service notices of MaxCompute in reverse chronological order.

Notice on the upgrade of exclusive resource groups of DataWorks in which PyODPS nodes run

Dear MaxCompute users:

The exclusive resource groups of DataWorks in which PyODPS nodes run will be upgraded in each region from March 11 (Monday), 2024 (UTC+8) to April 25 (Thursday), 2024 (UTC+8), to enhance support for PyODPS. The following table describes the upgrade plan in each region. After the upgrade, the package version of PyODPS nodes remains unchanged.

Item

Region

Upgrade date

  • Support for the load_resource_package method for PyODPS nodes of DataWorks

  • Support for multiple processes

  • Addition of automatic error prompts

China (Chengdu) and China (Zhangjiakou)

March 11, 2024 to March 14, 2024

China (Shenzhen)

March 18, 2024 to March 21, 2024

China (Beijing)

March 25, 2024 to March 29, 2024

China (Shanghai)

April 1, 2024 to April 5, 2024

China (Hangzhou)

April 8, 2024 to April 11, 2024

China (Hong Kong) and regions outside China

April 15, 2024 to April 18, 2024

China East 1 Finance and China East 2 Finance

April 22, 2024 to April 25, 2024

During the upgrade, the PyODPS jobs that are running may be affected. If a PyODPS job that uses an exclusive resource group fails or other related issues occur, contact technical support in your private DingTalk group or join the DingTalk group (ID: 11782920) for MaxCompute technical support.

Thank you for your cooperation.

February 28, 2024 (UTC+8): Notice on the drills for MaxCompute single-node failures

Dear MaxCompute users:

Drills for service access reliability are conducted in the China (Zhangjiakou) and China (Chengdu) regions from 15:00 to 16:00 on February 28, 2024 (UTC+8). In most cases, jobs are not affected by the drills. If your job fails due to the drills, rerun your job. This occurs at an extremely low probability.

If you have questions, contact technical support in your private DingTalk group or join the DingTalk group (ID: 11782920) for MaxCompute technical support.

Thank you for your cooperation.

January 25, 2024 (UTC+8): Notice on the release of the computing cost optimization feature of MaxCompute

Dear MaxCompute users:

The computing cost optimization feature is released on January 25 (Thursday), 2024 (UTC+8) to replace the computing resource optimization feature of MaxCompute.

The computing cost optimization feature is developed based on the computing resource optimization feature. The computing resource optimization feature can generate a resource configuration optimization plan for the computing resources in a level-1 subscription quota. Compared with the computing resource optimization feature, the computing cost optimization feature can also generate a resource configuration optimization plan for changing the pay-as-you-go computing resources of a project to subscription computing resources and allows you to view the estimated optimization effect. This helps you further optimize computing costs and improve resource utilization. For more information, see Generate a computing resource optimization plan.

If you have questions, contact technical support in your private DingTalk group or join the DingTalk group (ID: 11782920) for MaxCompute technical support.

Thank you for your cooperation.

December 28, 2023 (UTC+8): Notice on the upgrade of the endpoint access architecture of MaxCompute in the Alibaba Cloud public cloud

Dear MaxCompute users:

Thank you for your support for MaxCompute. The endpoint access architecture of MaxCompute is upgraded on January 4, 2024 (UTC+8) in the following regions: China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), China (Zhangjiakou), and China (Chengdu). After the upgrade, MaxCompute provides new endpoints. The physical hardware for the original endpoints cannot be upgraded or scaled out to meet the requirements for business growth due to the limits of the historical architecture. You can still use existing endpoints until the discontinuation date, which will be announced. We recommend that you switch the endpoints of your business at the earliest opportunity.

If you have questions, contact technical support in your private DingTalk group or join the DingTalk group (ID: 11782920) for MaxCompute technical support.

Thank you for your cooperation.

December 27, 2023 (UTC+8): Notice on the change of the traffic bandwidth for read and write operations on OSS external tables of MaxCompute to the tenant traffic bandwidth

Dear MaxCompute users:

Thank you for your support for MaxCompute. The architecture of the traffic bandwidth for read and write operations on Object Storage Service (OSS) external tables of MaxCompute will be upgraded on January 29, 2024 (UTC+8) in the China (Hangzhou) region and on February 29, 2024 in the China (Shanghai) and China (Beijing) regions. After the upgrade, the traffic bandwidth of tenants or the related buckets is used. Before the upgrade, the read and write operations on OSS external tables of MaxCompute use the public traffic bandwidth of MaxCompute in the China (Hangzhou), China (Shanghai), and China (Beijing) regions. The bandwidth isolation capability is weak and the traffic bandwidth cannot be monitored. Other regions are not involved in this issue. This upgrade improves the data access stability of OSS external tables, implements bandwidth isolation between tenants, and enables flexible configuration of bandwidth resources for OSS external tables.

If you have questions, contact technical support in your private DingTalk group or join the DingTalk group (ID: 11782920) for MaxCompute technical support.

Thank you for your cooperation.

December 27, 2023 (UTC+8): Notice on the termination of automatic installation of project-level Information Schema for new projects

Dear MaxCompute users:

Thank you for your support for MaxCompute. Tenant-level Information Schema of MaxCompute was fully released on August 1, 2023 (UTC+8), which allowed you to query the metadata of all projects under a tenant at a time. This improves query efficiency and completes the metadata based on tenant-level Information Schema. Therefore, we plan to terminate automatic installation of project-level Information Schema for new MaxCompute projects from March 1, 2024 (UTC+8). This indicates that new projects do not have the package of project-level Information Schema by default. If you want to query metadata, you can use tenant-level Information Schema to obtain more comprehensive information. For more information about how to use tenant-level Information Schema, see Tenant-level Information Schema.

If you have questions, contact technical support in your private DingTalk group or join the DingTalk group (ID: 11782920) for MaxCompute technical support.

Thank you for your cooperation.

December 15, 2023 (UTC+8): Notice on updates of lifecycle management policies of MaxCompute projects

Dear MaxCompute users:

Thank you for your support for MaxCompute. New lifecycle management policies will start to take effect for MaxCompute projects in different regions from December 25 (Monday), 2023 (UTC+8). If you use subscription computing resources of MaxCompute, you must take note of the policy update. The following content compares the lifecycle management policies before and after the update.

  • Before the update

    If you configure subscription computing resources as default computing resources for your MaxCompute project, the lifecycle of your project is aligned with the lifecycle of the subscription computing resources. For more information about how to configure default computing resources for a MaxCompute project, see Use of computing resources. After subscription computing resources expire, the project is suspended and unavailable. If you do not renew the project within 15 days, the project is deleted and cannot be restored. When you unsubscribe from subscription computing resources, the project is deleted and cannot be restored.

  • After the update

    • If the pay-as-you-go service is activated in the region to which your subscription computing resources belong, the following policies are applied. You can click pay-as-you-go to activate the pay-as-you-go service.

      • The lifecycle of a project in the region is not affected by the lifecycle of the subscription computing resources that are allocated to the project. When the subscription computing resources that are allocated to the project in the region expire or are released, only the associated quota is frozen or released. The status of the project in the region is not affected. This way, the project data is not deleted and can be normally requested by using other methods.

        Important

        If the quota of subscription computing resources is frozen or released and the default quota of the project is not changed to another normal quota, you cannot run computing jobs in the project. For more information about how to configure or specify a quota, see Use of computing resources.

      • The lifecycle of a project in the region is aligned with that of the pay-as-you-go service. If the project of your account has overdue payments, the project is frozen. If you do not settle the overdue payments within 15 days, the project is deleted and cannot be restored.

      • For the pay-as-you-go service bills generated for storage and download resource consumption of projects in the region, Product Details on the Billing Details tab is changed from MaxCompute (Subscription) to MaxCompute (Pay-As-You-Go), and Billable Item on the Usage Records tab is changed to MaxCompute(Postpay).

    • If the pay-as-you-go service is not activated in the region to which your subscription computing resources belong, the project lifecycle remains unchanged. In this case, we strongly recommend that you click pay-as-you-go to activate the pay-as-you-go service for free at the earliest opportunity. This prevents irreversible data loss when data is inaccessible or deleted due to expiration or release of subscription computing resources.

For more information about the project lifecycle, see Service deactivation and resource release.

Thank you for your cooperation.

December 4, 2023 (UTC+8): Notice on discontinuation of MaxCompute Management

Dear MaxCompute users:

Thank you for your support for MaxCompute. The new MaxCompute console was released in March 2023. It is integrated with capabilities provided by MaxCompute Management. The new MaxCompute console delivers a better user experience and provides more management and O&M capabilities. The new MaxCompute console has the basic capabilities provided by MaxCompute Management. MaxCompute Management will reach the end of maintenance and will be gradually discontinued on January 10 (Wednesday), 2024 (UTC+8).

Before MaxCompute Management is discontinued, you can still view information on MaxCompute Management. However, the edit operation is no longer allowed. The following capabilities provided by MaxCompute Management correspond to specific features of the new MaxCompute console:

  • Resource overview: provides a chart to show the trend of resource usage of a subscription quota. This feature corresponds to the Resource Observation feature in the new MaxCompute console. The Resource Observation feature allows you to view usage information of more resources. For more information about the Resource Observation feature, see Resource Observation.

  • Project O&M: allows you to view the project list. This feature corresponds to the Projects feature in the new MaxCompute console. For more information about the Projects feature, see Manage projects in the new MaxCompute console.

  • Quota management: allows you to manage level-2 quotas of a level-1 quota for subscription computing resources. This feature corresponds to the Quotas feature in the new MaxCompute console. In addition to all the quota management capabilities provided by MaxCompute Management, the Quotas feature supports elastically reserved CUs and quota management of exclusive resource groups for MaxCompute Tunnel. This significantly improves user experience. For more information about the Quotas feature, see Manage projects in the new MaxCompute console.

  • Job O&M: provides job snapshots and historical jobs. This feature corresponds to the Jobs feature in the new MaxCompute console. The Jobs feature supports all the job O&M capabilities provided by MaxCompute Management. For more information about the Jobs feature, see Manage jobs in the new MaxCompute console.

  • Optimization recommendations (supported in some regions): Data Governance in DataWorks supports this feature, including continuous empty output, continuous full table scan, and recommendations for enabling dynamic filters. For more information, see Output is empty and Violence Scan.

If you have questions, contact technical support in your private DingTalk group or join the DingTalk group (ID: 11782920) for MaxCompute technical support.

Thank you for your cooperation.

November 21, 2023 (UTC+8): Notice on configurations of the Super_Administrator role based on RAM permission control

Dear MaxCompute users:

From November 21 (Tuesday), 2023 (UTC+8), you can configure members of the Super_Administrator role in the MaxCompute console. To perform the configuration operation, find the project on the Projects page, and click Manage in the Actions column. On the Project Settings page, click the Parameter Configuration tab and configure members of the Super_Administrator role. For more information about the Super_Administrator role, see the "Built-in roles" section in Role planning. This operation supports RAM permission control. If a RAM user is granted the UpdateUsersToSuperAdmin permission, you can configure members of the Super_Administrator role for a project as the RAM user. This way, you can manage the project by using the permissions of the Super_Administrator role. For more information, see RAM permissions. This feature is suitable for the following scenario: The project owner uses an account in a resource directory and cannot access MaxCompute for authorization. In this scenario, you cannot configure the parameters that require the permissions of the Super_Administrator role.

Thank you for your cooperation.

November 16, 2023 (UTC+8): Notice on the availability of elastically reserved CUs in MaxCompute

Dear MaxCompute users:

Elastically reserved CUs will be available on November 16 (Thursday), 2023 (UTC+8).

If you purchase subscription reserved CUs, you can also use elastically reserved CUs to perform scheduled auto scaling. Elastically reserved CUs are billed on a pay-as-you-go basis. You are charged for elastically reserved CUs based on the usage duration and number of elastically reserved CUs. For the specifications and unit price of elastically reserved CUs, see Computing pricing (hourly billing). After you purchase subscription reserved CUs, you can log on to the new MaxCompute console and choose Workspace > Quotas. On the Quotas page, find the level-1 quota and click Quota Configuration in the Actions column. On the Quota Plans tab of the Quota Configuration page, you can configure elastically reserved CUs for the level-1 quota and use elastically reserved CUs with time plans to perform scheduled auto scaling. For more information about how to configure elastically reserved CUs, see Manage quotas in the new MaxCompute console.

Thank you for your cooperation.

September 25, 2023 (UTC+8): Notice on billing of SQL subqueries and change on the maximum job running duration in MaxCompute

Dear MaxCompute users:

From December 1 (Friday), 2023 (UTC+8), the following SQL behaviors in MaxCompute will be changed. The change applies to the customers in regions outside the Chinese mainland, customers in Finance Cloud regions, and customers in Alibaba Gov Cloud regions. The customers need to learn about relevant information in advance.

  • The billing issue of subqueries is fixed. In earlier versions that use the pay-as-you-go billing method, IN subqueries in SQL statements are not billed. This issue is fixed. Since 10:00:00 on December 1, 2023, you are charged for IN subqueries.

  • The default maximum running duration of an SQL job is changed from 72 hours to 24 hours. This change prevents system resources from being occupied by invalid jobs for a long period of time. The change takes effect at 10:00:00 on December 1, 2023. If you want to run a job for more than 24 hours, run the set odps.sql.job.max.time.hours=72; command to change the maximum running duration for the job before the change takes effect. The maximum running duration that you can configure is 72 hours.

Thank you for your cooperation.

September 21, 2023 (UTC+8): Notice on the release of the job analysis feature in MaxCompute

Dear MaxCompute users:

From September 21 (Thursday), 2023 (UTC+8), the job analysis feature of MaxCompute is released in Alibaba Cloud regions in the Chinese mainland.

The job analysis feature allows you to view the resource consumption of a specified job and the resource allocation of computing quotas of a specified job at a specific point in time on the Jobs page. For more information, see Manage jobs in the new MaxCompute console. If a job is not complete for a long period of time or it is found that a job runs at a low speed after the job is complete, you can check whether this issue occurs due to insufficient computing resources or multi-job resource preemption based on the resource consumption data obtained from the job analysis result. Then, you can adjust job priorities or manage computing resources to optimize job execution based on your business requirements. For more information, see Job priority or the "Configure quotas" section in Manage quotas in the new MaxCompute console.

For more information about how to view the resource consumption of a job, see the "View the resource consumption of a job and the resource allocation of computing quotas at a specific point in time" section in Manage jobs in the new MaxCompute console.

Thank you for your cooperation.

August 8, 2023 (UTC+8): Notice on the availability of MaxCompute tenant-level Information Schema

Dear MaxCompute users:

Tenant-level Information Schema provides information such as project metadata and usage history data from the perspective of tenants. Tenant-level Information Schema allows you to pull a specific type of metadata from all projects of a tenant in the same metadata center at a time. You do not need to query metadata from each project. This delivers higher query efficiency than project-level Information Schema. If the tenant-level Information Schema service is activated for your Alibaba Cloud account, a project named SYSTEM_CATALOG is automatically created for the Alibaba Cloud account, and a built-in schema named INFORMATION_SCHEMA is provided. You can access the read-only views that are provided by the built-in schema to query the metadata and usage history of all projects of your Alibaba Cloud account. For more information, see Tenant-level Information Schema.

Thank you for your cooperation.

July 3, 2023 (UTC+8): Notice on a free trial of MaxCompute Transaction Table 2.0

Dear MaxCompute users:

The Transaction Table 2.0 feature of MaxCompute is available for a free trial from July 3 (Monday), 2023 (UTC+8).

The Transaction Table 2.0 feature of MaxCompute is an enhanced version of the Transaction Table feature. The Transaction Table 2.0 feature allows you to configure primary keys for tables and supports high-concurrent UPSERT or DELETE write operations on tables within minutes. The Transaction Table 2.0 feature also supports the time travel feature and incremental queries and meets the requirements for read and write snapshot isolation. For more information about the Transaction Table 2.0 feature and how to use the feature, see Overview of Transaction Table 2.0.

The schema feature of MaxCompute is available in all regions. For more information about how to apply for a free trial, see Apply for trial use of new features.

Thank you for your cooperation.

July 3, 2023 (UTC+8): Notice on a free trial of JSON data types in MaxCompute

Dear MaxCompute users:

JSON data types in MaxCompute are available for a free trial from July 3 (Monday), 2023 (UTC+8).

MaxCompute supports JSON data types. When JSON data is inserted into a table, MaxCompute automatically performs data optimization and uses column-oriented storage to increase the compression ratio of JSON data and improve the query and analysis performance. For more information about JSON data types and how to use JSON data types, see User guide for JSON data types in MaxCompute (public preview).

This feature is available in all regions. If you want to use this feature together with other cloud services, you must make sure that the cloud services support JSON data types.

Thank you for your cooperation.

May 10, 2023 (UTC+8): Notice on the upgrade of MaxCompute

Dear MaxCompute users:

To improve network disaster recovery capabilities, MaxCompute in the following regions will be upgraded during the period from 15:00:00 to 16:00:00 on May 24 (Wednesday), 2023 (UTC+8): China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), and China (Chengdu).

During the upgrade, network settings are changed at the access layer of MaxCompute. A records are added to domain names. In normal cases, you can use MaxCompute during this period of time. However, if network connections are established by using a Cloud Enterprise Network (CEN) or a virtual private cloud (VPC) and the virtual IP address of MaxCompute is configured in the route table, you need to contact Alibaba Cloud technical support engineers at the earliest opportunity and ask them to add the new virtual IP address to the route table.

We apologize for any inconvenience caused. If you have any questions, submit a ticket or contact the service hotline.

Thank you for your cooperation.

April 6, 2023 (UTC+8): Notice on the release of the job management feature in the new MaxCompute console

Dear MaxCompute users:

MaxCompute will release a new feature to manage jobs in the new MaxCompute console from April 6 (Thursday), 2023 (UTC+8). If you enable this feature, you can obtain a complete list of MaxCompute jobs and manage the jobs in the new MaxCompute console. If you are a data developer, you can view and terminate MaxCompute jobs. If you are an administrator, you can view the resource usage at a specific time point based on quota groups and terminate MaxCompute jobs.

In the new MaxCompute console, you can use the job management feature to view the job snapshot information and job information. You can also perform the following operations after you specify the start time and end time for the Time Range parameter:

  • Filter out the jobs that are completed within the specified time range.

  • Filter out the jobs that are running at the end time or 3 minutes before the end time. The job snapshot information is displayed. The period of time is also called the job snapshot observation time.

For more information about the job management feature, see Manage jobs in the new MaxCompute console.

Thank you for your cooperation.

March 28, 2023 (UTC+8): Notice on the release of the new MaxCompute console

Dear MaxCompute users:

MaxCompute will release a new MaxCompute console on March 28 (Tuesday), 2023 (UTC+8). The new MaxCompute console uses a new endpoint. Features of the new MaxCompute console will be first released in the Chinese mainland regions. In the regions in which the new MaxCompute console is not available, you can log on to the old MaxCompute console by using the new endpoint.

The new MaxCompute console provides new capabilities in project management and quota management and is integrated with new features such as elastically reserved CUs and custom Admin-type roles at the project level.

  • Comparison between the new and old MaxCompute consoles

    • Access control

      Before you log on to the new MaxCompute console to perform management operations in MaxCompute, you must make sure that related policies are attached to your account. For more information, see RAM permissions.

    • Project management

      • Project creation

        In the old MaxCompute console, a MaxCompute project is created when you create a DataWorks workspace. In the new MaxCompute console, you can separately create a MaxCompute project. We recommend that you continue to create and use MaxCompute projects in the DataWorks console.

      • Project deletion

        In the old MaxCompute console, you can delete MaxCompute projects with which DataWorks workspaces are associated. You must disassociate the MaxCompute projects from the DataWorks workspaces before you delete the MaxCompute projects. Both logical deletion and immediate deletion of MaxCompute projects are supported. A MaxCompute project can be restored after it is logically deleted. In the new MaxCompute console, you can delete all MaxCompute projects only by using the immediate deletion method. Before you delete a MaxCompute project in the new MaxCompute console, you must confirm that the MaxCompute project has been disassociated from the DataWorks workspace. This ensures that the DataWorks workspace runs as expected after the MaxCompute project is deleted.

      • Project status change

        In the old MaxCompute console, the project status changes with the lifecycle of the purchased instance. You can customize the status change for logical deletion and restoration of projects. In the new MaxCompute console, the project status also changes with the lifecycle of the purchased instance. You can configure parameters to disable and restore projects based on your business requirements.

      • Default quota switching of projects

        In the old MaxCompute console, you can update the quota configurations of the MaxCompute projects with which DataWorks workspaces are associated and switch the used quota to the default quota in the DataWorks console. If a DataWorks workspace works in standard mode, the quota configurations of the two projects with which the workspace is associated are updated at the same time. In the new MaxCompute console, you can separately update the quota configurations of each project that is in the normal or suspended state.

      • Project parameter settings

        In the old MaxCompute console, you cannot configure parameters for projects. In the new MaxCompute console, you can configure common project information such as basic project attributes, permission attributes, and IP address whitelists. You can perform related operations in the new MaxCompute console by using an account that is granted related permissions.

      • Project permission management

        In the old MaxCompute console, you cannot manage project permissions. In the new MaxCompute console, you can manage project roles and packages and manage permissions based on roles. The verification of management permissions in the new MaxCompute console is the same as the verification of management permissions based on SQL statements.

        • Project roles: You can create Resource-type roles or Admin-type roles. ACL-based access control and policy-based access control are supported for role authorization. In the new MaxCompute console, ACL-based access control allows you to grant permissions on all MaxCompute objects, such as tables, functions, resources, packages, and projects. Batch authorization is supported.

        • Packages: You can perform operations on packages in the new MaxCompute console. For example, you can create a package, grant or revoke permissions on a package, or install a package. Batch authorization is supported.

        Note

        The role management feature of MaxCompute projects will be no longer available on the Project management tab in the DataWorks console. The permission verification logic provided by the role management feature is inconsistent with the permission verification logic used in the new MaxCompute console. If your MaxCompute service is not deployed in the Chinese mainland regions and you must use a RAM account or a RAM role to manage permissions, you can configure parameters for permission management of MaxCompute projects in the DataWorks console before the role management feature is unavailable. This way, you can use the RAM account or RAM role to manage projects in the new MaxCompute console.

    • Quota management

      Note

      Quota management is available only in the China (Shanghai), China (Hangzhou), China (Beijing), China (Shenzhen), China (Zhangjiakou), and China (Chengdu) regions. You are notified if more regions are supported.

      The new MaxCompute console is integrated with the quota management feature that is provided in MaxCompute Management.

      • Quota lists

        Level-1 quotas and level-2 quotas are displayed. The parameter properties of the current quota are displayed.

      • Resource consumption

        You can view the trend of CU consumption of a level-1 quota. The displayed trend is consistent with the trend that is displayed in MaxCompute Management.

      • Upgrade, downgrade, and renewal

        You can perform operations on level-1 quotas. For example, you can upgrade, downgrade, or renew level-1 quotas of MaxCompute instances.

      • Quota configurations

        You can configure level-2 quotas for each level-1 quota and schedule different quotas based on time plans.

        • Quota template: allows you to create and delete level-2 quotas.

        • Quota plan: allows you to globally configure quota properties and the number of CUs. You can configure parameters to schedule different quotas based on time plans.

        • Time plan: allows you to configure time periods during which different quotas are scheduled in a day. Quota scheduling based on time plans can only start on the hour or half-hour.

          If the time-based quota feature is enabled in MaxCompute Management, quota plans are generated for each time period in the new MaxCompute console. Quota plans are named in the p_Start time format. You can use time plans to configure the time periods in which different quota plans are enabled.

  • Custom Admin-type roles at the project level

    In the new MaxCompute console, you can click Projects in the left-side navigation pane, click Manage in the Actions column of your project, and then click the Role Permissions tab to create an Admin-type role. You can use only the policy-based access control method to perform authorization operations for Admin-type roles. Admin-type roles allow you to customize the scope of permissions for security administrators. For example, you can authorize an Admin-type role to configure an IP address whitelist of a project, or authorize an Admin-type role to manage data permissions of a project.

Thank you for your cooperation.

March 11, 2023 (UTC+8): Notice on the adjustments of MaxCompute PyODPS

Dear MaxCompute users:

The following adjustments are applied to PyODPS V0.11.3 and later in MaxCompute from March 11 (Saturday), 2023 (UTC+8).

  • MaxCompute introduces a new concept called schema. You can classify objects by schema. For more information about schema operations, see Schema-related operations. The schema name conflicts with the schema field. We recommend that you use the table_schema field to specify a table schema. If you use the schema field to specify a table schema, an alert is reported, but the schema usage is not affected. The following code shows an example:

    --# Obtain the schema of a table. 
    t = o.get_table('pyodps_iris')
    print (t.table_schema)  # Obtain the schema of the pyodps_iris table.
  • By default, time fields for resources and functions, such as creation_time, use the local time zone instead of the UTC time zone. This way, the time zone used by time fields for resources and functions is the same as the time zone used by time fields for tables and partitions. If you want to use the UTC time zone, configure the following settings:

    options.use_legacy_parsedate = True
  • The time field last_modified_time of the original table and partitions in the table is changed to last_data_modified_time. If you use the original field name, an alert is reported, but the schema usage is not affected.

Thank you for your cooperation.

January 18, 2023 (UTC+8): Notice on restrictions on the maximum number of slots that users can use in MaxCompute Tunnel shared resource groups

Dear MaxCompute users:

To provide a more stable resource sharing service in MaxCompute Tunnel, MaxCompute will apply restrictions on the maximum number of slots that users can use in Tunnel shared resource groups in each region from March 1 (Wednesday), 2023 (UTC+8).

MaxCompute provides Tunnel shared resource groups for users in each region to upload and download data. If the resource usage of a specific user in a region surges, a large number of Tunnel shared resources may be occupied by the user. As a result, some users may fail to obtain shared resources. The restrictions are applied to resolve the issue. For more information, see Data upload scenarios and tools.

Thank you for your cooperation.

January 9, 2023 (UTC+8): Notice on fixing a bug of incorrect billing on storage resources of subscription MaxCompute services

Dear MaxCompute users:

Due to a billing configuration exception of storage resources of subscription MaxCompute services, the storage fees that are actually deducted for subscription MaxCompute services are lower than the fees that are calculated based on the billing mechanism of the official website. The bug does not affect the use of storage resources of subscription MaxCompute services. The bug will be fixed on January 18, 2023 (UTC+8). After the bug is fixed, the actual storage fees are the same as the fees that are calculated based on the billing mechanism of the official website. You can view billing details in your bills. For more information about storage billing rules of subscription MaxCompute services, see Storage pricing (pay-as-you-go).

October 13, 2022 (UTC+8): Notice on a free trial of the schema feature in MaxCompute

Dear MaxCompute users:

As of October 13 (Thursday), 2022 (UTC+8), the schema feature of MaxCompute is available for a free trial.

MaxCompute provides the schema feature. A schema is created in a project. You can use schemas to classify objects in projects, such as tables, resources, and user-defined functions (UDFs). You can create multiple schemas in a project to store data by business category. For more information about the schema feature and how to use schemas, see Schema-related operations.

The schema feature of MaxCompute is available in all regions. For more information about how to apply for a free trial, see Apply for trial use of new features.

Thank you for your cooperation.

October 10, 2022 (UTC+8): Notice on the new policy for the expiration of subscription MaxCompute services

Dear MaxCompute users:

A new policy for the expiration of subscription MaxCompute services at the international site (alibabacloud.com) will be used from October 10 (Monday), 2022 (UTC+8).

Original policy: If you do not renew your MaxCompute service within 15 days after the service expires, the service is suspended. Resources are released on the 15th day after the service is suspended. After the resources are released, the data of all MaxCompute projects is lost and cannot be recovered.

New policy: If you do not renew your MaxCompute service after the service expires, the service is immediately suspended and resources are released on the 15th day. After the resources are released, the data of all MaxCompute projects is lost and cannot be recovered.

After the new policy takes effect, MaxCompute still reminds you to renew your MaxCompute service seven days, three days, and one day before the service expires. We recommend that you enable auto-renewal to ensure service continuity. For more information about how to enable auto-renewal, see the "Enable automatic renewal for an instance" section in Renewal management.

Thank you for your cooperation.

August 30, 2022 (UTC+8): Notice on a routine update of the MaxCompute client odpscmd

Dear MaxCompute users:

The update operation will be performed on the MaxCompute client odpscmd during the period from 10:30:00 to 11:00:00 on August 30 (Tuesday), 2022 (UTC+8) for MaxCompute projects in the China (Hong Kong) and Singapore regions. The update operation will be performed on the MaxCompute client odpscmd during the period from 10:30:00 to 11:00:00 on August 31 (Wednesday), 2022 (UTC+8) for MaxCompute projects in the China East 2 Finance, China North 2 Ali Gov 1, and China South 1 Finance. If you submit a MaxCompute task by using DataWorks during the update, errors that indicate Java class loading exceptions, such as ClassNotFound and NoClassDefFoundError, may occur with a small probability. If these errors occur, retry the task. We are sorry for the inconvenience caused by this update.

August 24, 2022 (UTC+8): Notice on an update of the SAMPLE function

Before the SAMPLE function is updated, results are returned as expected if you do not specify a hash column in the SAMPLE function, such as sample(10,1). However, if you specify a hash column in the SAMPLE function, the number of entries in the returned results may be invalid. If the sampled data is of the BIGINT type, such as 10187140, 10189550, and 1018910, True is always returned. This is not the expected result.

The updated SAMPLE function optimizes the hash effect of the generated results by correcting the final processing method of the hash algorithm. This prevents uneven data distribution. The results that are returned by a SAMPLE function before and after the update may be different. This issue does not affect the accuracy of the returned results. Example:

-- Execute the following SQL statement:
select count(*) from (values (10024350), (14564560), (12243560)) as X(a) where sample(10, 1, a)=true;

-- Returned result before the update:
+------------+
| _c0        |
+------------+
| 0          |
+------------+

-- Returned result after the update:
+------------+
| _c0        |
+------------+
| 1          |
+------------+

For more information about the SAMPLE function, see the "SAMPLE" section in Other functions.

August 11, 2022 (UTC+08:00): MaxCompute V1.0 compatibility mode is discontinued

Dear MaxCompute users:

As of July 2017, the MaxCompute team continuously optimizes MaxCompute V2.0 to enhance user experience and provide better service performance.

MaxCompute V2.0 has been working stably for about five years. MaxCompute V1.0 compatibility mode is discontinued at 24:00:00 on December 20, 2022 (UTC+8) based on the product lifecycle.

MaxCompute V2.0 supports more strict syntax checks than MaxCompute V1.0. As a result, in MaxCompute V2.0, errors may be returned for some statements that can be successfully executed in MaxCompute V1.0. For more information about the errors and solutions, see Error messages and solutions.

We recommend that you check your SQL statements and modify the statements that are not standard in a timely manner before MaxCompute V1.0 compatibility mode is discontinued. You can add set odps.sql.planner.mode=lot; before your SQL statement to set an SQL flag at the session level to check whether the SQL statement can be executed in MaxCompute V2.0.

If you have any questions, contact technical support in your private DingTalk group or join the DingTalk group for MaxCompute technical support. You can scan the following QR code to join the DingTalk group for MaxCompute technical support.MaxCompute开发者钉钉群

Thank you for your support for MaxCompute.

November 29, 2021 (UTC+8): The VPC connection scheme is released for commercial use

MaxCompute can use external tables, user-defined functions (UDFs), or the data lakehouse solution to access a virtual private cloud (VPC). Before MaxCompute accesses a VPC, you must establish a network connection between MaxCompute and the destination IP address or service, such as ApsaraDB for HBase, ApsaraDB RDS, or Hadoop clusters.

The VPC connection scheme is released for commercial use on November 29, 2021 (UTC+8) in the China (Shanghai), China (Hangzhou), China (Beijing), China (Shenzhen), and China (Zhangjiakou) regions. The VPC connection scheme allows you to manage network connections and grant permissions to administrators by using an Alibaba Cloud account or a RAM user that is assigned the tenant-level Super_Administrator or Admin role. You can manage network connections and grant permissions to administrators on the Network Links and Users tabs in the MaxCompute console. For more information about how to connect MaxCompute to a VPC, see Network connection process. For more information about how to grant permissions to administrators, see Role planning and the "Assign a role to a user" section in Perform access control based on project-level roles.

November 1, 2021 (UTC+8): MaxCompute Query Acceleration (MCQA) is released for commercial use

MaxCompute creates a temporary dataset for each MCQA query job to cache the query results. If you run the same query job, the system first returns the results that are cached in the temporary dataset. The cached query results are maintained based on query jobs. The cached query results do not generate storage or computing fees. Therefore, MCQA reduces your costs and significantly speeds up queries.

October 15, 2021 (UTC+8): Notice on the maintenance of MaxCompute Management

MaxCompute Management is maintained from 22:00 to 24:00 on October 15, 2021 (UTC+8). During this period, the features provided by MaxCompute Management, such as quota management and job maintenance, become unavailable. You can use MaxCompute Management after the maintenance is completed.

October 15, 2021 (UTC+8): The backup and restoration feature is released for commercial use

From October 15, 2021, you are charged for data that is backed up on MaxCompute on a pay-as-you-go basis.

Take note of the following billing rules:

  • MaxCompute projects automatically back up data after changes and generate a data version for each change. By default, all data versions are retained for one day free of charge.

  • A project administrator can specify a longer retention period for backup data. Backup data that is retained for more than one day is charged based on the volume of the data. The unit price for backup data is USD 0.0006 per GB per day.

For more information about the backup and restoration feature, see Backup and restoration. For more information about storage pricing, see Storage pricing (pay-as-you-go).

September 10, 2021 (UTC+8): The UPDATE, DELETE, and MERGE INTO statements are released for commercial use

From September 10, 2021, the public preview of the UPDATE, DELETE, and MERGE INTO statements of MaxCompute SQL ends. After the public review ends, you are charged when you use these statements. Pay-as-you-go jobs that already use the UPDATE, DELETE, and MERGE INTO statements are charged after the public preview ends. If you did not use the UPDATE, DELETE, or MERGE INTO statement due to concerns about service stability in the public preview phase, you no longer need to worry about stability when you use these statements. MaxCompute provides the same guarantees for the availability and stability of the UPDATE, DELETE, and MERGE INTO statements as the guarantee for MaxCompute SQL.

Take note of the following billing rules:

  • If the billing method of your MaxCompute project is subscription, you are not separately charged for data computations.

  • If the billing method of your MaxCompute project is pay-as-you-go, you are charged for data computations. The computing fee is calculated by using the following formula: Computing fee of a single SQL statement = Volume of input data × SQL complexity × Unit price 0.0438 USD/GB. The SQL complexity is measured based on the number of SQL keywords. Number of SQL keywords = Number of JOIN clauses + Number of GROUP BY clauses + Number of ORDER BY clauses + Number of DISTINCT clauses + Number of window functions + MAX(Number of INSERT statements|Number of UPDATE statements|Number of DELETE statements - 1, 1).

To prevent excessive costs, you can configure alerts to control costs. For more information about monitoring and alerting, see Consumption control.

Take note of the following points:

  • If the billing method of a MaxCompute project is pay-as-you-go, MaxCompute calculates fees by using the following formula: Volume of input data scanned by using the UPDATE, DELETE, and MERGE INTO statements × SQL complexity × Unit price. When you execute an UPDATE statement or a DELETE statement on a transactional table, a Delta file is generated. If a large number of Delta files are generated, the data expands and the read performance is affected. In addition, the amount of data scanned increases gradually, which increases costs. A MaxCompute project that uses the subscription billing method occupies more resources and decreases the efficiency of subsequent job execution. You can run the compact command to merge Delta files. For more information about the syntax, see the "ALTER TABLE COMPACT" section in UPDATE and DELETE.

  • MaxCompute can execute multiple DELETE and UPDATE statements at a time. Each statement uses resources and incurs fees. We recommend that you delete or update a batch of data at a time. For more information, see UPDATE and DELETE. Sample command:

    -- Recommended: Use the association method to update or delete all the required data at the same time. 
    update table1 set col1= (select value1 from table2 where table1.id = table2.id and table1.region = table2.region);
    
    -- Not recommended: Perform related operations on the data row by row. 
    update table1 set col1=1 where id='2021063001'and region='beijing';                  
    update table1 set col1=2 where id='2021063002'and region='beijing';
    ......

April 20, 2021 (UTC+8): The ON clause of OUTER JOIN does not filter data for new MaxCompute projects

Before the update, the filter condition specified for a single table in the ON clause filters data before a JOIN operation is performed, regardless of whether the JOIN operation is OUTER JOIN. For example, the execution results of Script 1, Script 2, and Script 3 in the following code are consistent.

-- Construct data. 
CREATE TABLE t1 AS SELECT * FROM VALUES (1, 20180101),(2, 20180101),(2, 20180102) t (key, ds);
CREATE TABLE t2 AS SELECT * FROM VALUES (1, 20180101),(3, 20180101),(2, 20180102) t (key, ds);

-- Script 1: The filter condition is in the ON clause. 
SELECT t1.*,t2.* 
FROM t1 
LEFT OUTER JOIN t2 
ON t1.key = t2.key and t1.ds='20180101';

-- Script 2: The filter condition is in the subquery. 
SELECT a.*,t2.* 
FROM (SELECT * FROM t1 WHERE t1.ds='20180101') a 
LEFT OUTER JOIN t2 
ON a.key = t2.key;

-- Script 3: The filter condition is in the WHERE clause. 
SELECT t1.*,t2.* 
FROM t1 
LEFT OUTER JOIN t2 
ON t1.key = t2.key 
WHERE t1.ds='20180101';

This is not a standard behavior. If values that are not 20180101 are included in the t1.ds field in Script 1, LEFT OUTER JOIN retains all the values of the left table, performs a left join on the left table t1 and the right table t2, and returns the data in the matched rows or returns NULL for rows in which no data is matched. This is the standard processing behavior. However, in Script 2 and Script 3, values that are not 20180101 in the t1.ds field are filtered out and are not displayed in the output. The execution results of Script 2 and Script 3 are inconsistent with the execution result of Script 1.

The default processing behavior in MaxCompute is changed to support the standard processing behavior. If you want to filter data from table t1, use the processing behavior of Script 2 or Script 3.

For MaxCompute projects that are created on or after April 20, 2021, the filter condition in the ON clause does not filter data before the OUTER JOIN operation is performed. For MaxCompute projects that are created before April 20, 2021, the original processing behavior is retained to prevent the impact of the update on existing jobs. For existing MaxCompute projects, we recommend that you use the new processing method. Setting method:

  • Session level: Add the set odps.task.sql.outerjoin.ppd=true; command before an SQL statement and commit the SQL statement with the command.

  • Project level: Contact MaxCompute technical support engineers in the DingTalk group to change the value of odps.task.sql.outerjoin.ppd to True for your project.

The following example helps you determine which processing behavior is used in your MaxCompute project.

-- Construct data based on the preceding code. 
SELECT t1.*, t2.* 
FROM t1 FULL OUTER JOIN t2 
ON t1.key = t2.key and t1.ds='20180101' and t2.ds='20180101';

-- The following results are returned if the original processing behavior is used. 
| key        | ds         | key2       | ds2        |
+------------+------------+------------+------------+
| 2          | 20180101   | NULL       | NULL       |
| 1          | 20180101   | 1          | 20180101   |
| NULL       | NULL       | 3          | 20180101   |

-- The following results are returned if the new processing behavior is used. 
| key        | ds         | key2       | ds2        |
+------------+------------+------------+------------+
| NULL       | NULL       | 2          | 20180102   |
| 2          | 20180101   | NULL       | NULL       |
| 2          | 20180102   | NULL       | NULL       |
| 1          | 20180101   | 1          | 20180101   |
| NULL       | NULL       | 3          | 20180101   |

April 20, 2021 (UTC+8): The GET_JSON_OBJECT function that is used in new projects retains the original strings instead of using escape characters to return JSON-formatted results

Before the update, the GET_JSON_OBJECT function returns JSON-formatted results by using escape characters. In this case, reserved characters in the results, such as line feeds (\n) and quotation marks ("), are presented as '\n' or '\'''. These formats are not standard.

After the update, the GET_JSON_OBJECT function retains the original strings. This ensures standard returned results and better data processing performance. After the update, the returned results are still JSON strings. However, you no longer need to use functions such as REGEXP_REPLACE or REPLACE to replace backslashes (\).

From April 20, 2021, the GET_JSON_OBJECT function automatically retains the original strings for the returned results in new MaxCompute projects. For MaxCompute projects created before April 20, 2021, the GET_JSON_OBJECT function automatically returns JSON-formatted reserved characters by using escape characters. This prevents the impact on existing jobs.

You can run the following command to determine the method that the GET_JSON_OBJECT function uses to return results in a MaxCompute project:

select get_json_object('{"a":"[\\"1\\"]"}', '$.a');
-- Return JSON reserved characters by using escape characters.
[\"1\"]

-- Return the original strings.
["1"]

We recommend that you use the new method. You can use this method at the following levels:

  • Session level: Add the set odps.sql.udf.getjsonobj.new=true; command before an SQL statement and commit the SQL statement with the command.

  • Project level: Contact MaxCompute technical support engineers in the DingTalk group to change the method that the GET_JSON_OBJECT function uses to return results in your MaxCompute project.

For more information about the GET_JSON_OBJECT function, see String functions.

December 14, 2020 (UTC+8): Check on input parameters of some date functions in MaxCompute is changed

MaxCompute strictly checks the format of input parameters of some date functions to prevent inaccurate results or other issues caused by invalid input. This change applies to all regions from December 14, 2020 to January 30, 2021 at the international site (alibabacloud.com) where MaxCompute is available. You must evaluate the current usage of related functions in your project and prepare for the update in advance. The update involves the following functions:

  • CAST(expr AS DATE)

    • Before the update: The CAST(expr AS DATE) function does not strictly check input strings, and characters that are not of the DATE type may be used as the input parameters. For example, execute the following query statement:

      select cast(a as date) from values ('2020-1x-19') t(a);

      The following code shows a sample query result:

      +------------+
      | a          |
      +------------+
      | 2020-01-19 |
      +------------+
    • After the update: The CAST(expr AS DATE) function strictly checks the input strings. Only the input strings whose year, month, and day fields are all represented by digits can pass the check. If one of the three fields in an input string contains letters or spaces, the CAST(expr AS DATE) function considers the input string invalid and returns NULL. For example, execute the following query statement:

      select cast(a as date) from values ('2020-1x-19') t(a);

      The following code shows a sample query result:

      +------------+
      | a          |
      +------------+
      | NULL       |
      +------------+
  • TO_DATE

    • Before the update: The TO_DATE function does not strictly check the input strings, and characters that are not of the DATE type may be used as the input parameters. For example, execute the following query statement:

      select to_date(a) from values ('2020-1x-19') t(a);

      The following code shows a sample query result:

      +------------+
      | _c0        |
      +------------+
      | 2020-01-19 |
      +------------+
    • After the update: The TO_DATE function strictly checks input strings. Only the input strings whose year, month, and day fields are all represented by digits can pass the check. If one of the three fields in an input string contains letters or spaces, the TO_DATE function considers the input string invalid and returns NULL. For example, execute the following query statement:

      select to_date(a) from values ('2020-1x-19') t(a);

      The following code shows a sample query result:

      +------------+
      | _c0        |
      +------------+
      | NULL       |
      +------------+
  • CAST(expr AS TIMESTAMP)

    • Before the update: The CAST(expr AS TIMESTAMP) function does not strictly check input strings, and characters that are not of the DATE type may be used as the input parameters. For example, execute the following query statement:

      select cast(a as timestamp) from values ('2020-11-19 16:3x:00.001') t(a);

      The following code shows a sample query result:

      +------------+
      | a          |
      +------------+
      | 2020-11-19 16:03:00.1 |
      +------------+
    • After the update: The CAST(expr AS TIMESTAMP) function strictly checks the input strings. Only the input strings whose year, month, and day fields are all represented by digits can pass the check. If one of the three fields in an input string contains letters or spaces, the CAST(expr AS TIMESTAMP) function considers the input string invalid and returns NULL. For example, execute the following query statement:

      select cast(a as timestamp) from values ('2020-11-19 16:3x:00.001') t(a);

      The following code shows a sample query result:

      +------------+
      | a          |
      +------------+
      | NULL       |
      +------------+

October 13, 2020 (UTC+8): The SQL engine is updated for better compatibility

The following table describes the schedule for updating the SQL engine. If any change occurs, the new schedule prevails.

Sequence

Region

Upgrade duration

1

India (Mumbai), Indonesia (Jakarta), and UK (London)

October 13, 2020

2

US (Virginia), UAE (Dubai), China North 2 Ali Gov, and China East 2 Finance

October 15, 2020

3

Japan (Tokyo), Australia (Sydney), US (Silicon Valley), and Malaysia (Kuala Lumpur)

October 20, 2020

4

Singapore, China (Hong Kong), and Germany (Frankfurt)

October 22, 2020

The URL_DECODE and CONV functions in MaxCompute SQL are updated. The following content describes the update details.

  • URL_DECODE

    • Before the update: If two parameters are specified for the URL_DECODE function, the function ignores the second parameter and decodes the value of the first parameter in UTF-8. For example, if you specify URL_DECODE(url, "gbk") in code, URL_DECODE(url) is executed.

    • After the update: If two parameters are specified for the URL_DECODE function, the function first performs decoding based on the percent sign (%). Then, the function performs decoding based on the encoding format of the second parameter. The following examples show how the URL_DECODE function works in DataWorks DataStudio before and after the function is updated:

      SELECT URL_DECODE("%CD%F5", "gbk");
      -- Before the update, the function returns garbled characters. gbk specified in the preceding SELECT statement does not take effect. UTF-8 is still used for decoding. 
      -- After the update, the function returns the Chinese character 王. \xCD\xF5 is a GBK-encoded string of the Chinese character 王. 
      
      SELECT URL_DECODE("%E7%8E%8B", "gbk");
      -- Before the update, the function returns the Chinese character 王. %E7%8E%8B is a UTF-8-encoded string of the Chinese character 王. gbk specified in the preceding SELECT statement does not take effect. UTF-8 is still used for decoding. 
      -- After the update, the function returns NULL. \xE7\x8E\x8B is an invalid GBK-encoded string. 
      
      SELECT URL_DECODE("%E7%8E%8B");
      -- The function returns the Chinese character 王 both before and after the update. %E7%8E%8B is a UTF-8-encoded string of the Chinese character 王. In this example, no decoding format is specified for the function, and the function decodes %E7%8E%8B in UTF-8 by default.

      Note

      The command-line interface (CLI) of the Windows operating system has the following issue: If you use the MaxCompute client (odpscmd) to run the cmd command in the Windows operating system, the GBK decoding result of the URL_DECODE function may be decoded in another format.

  • CONV

    • In a project that uses the Hive-compatible data type edition, the CONV function returns 0 both before and after the update if the input parameters are invalid.

    • In a project that uses the MaxCompute V1.0 or MaxCompute V2.0 data type edition:

      • Before the update: The CONV function returns garbled characters if the input parameters are invalid.

      • After the update: The CONV function returns NULL if the input parameters are invalid.

        For example, if you specify CONV("00e04c9d034a", 2, 10) in code, NULL is returned.

July 24, 2020 (UTC+8): MaxCompute provides more built-in aggregate functions

The following built-in aggregate functions are added to MaxCompute:

  • APPROX_DISTINCT(value): returns the approximate number of distinct input values.

  • ANY_VALUE(value): returns a non-deterministic value from the specified value range.

  • ARG_MAX(valueToMaximize, valueToReturn): finds the row where the maximum value of valueToMaximize resides and then returns the value of valueToReturn in the row.

  • ARG_MIN(valueToMinimize, valueToReturn): finds the row where the minimum value of valueToMinimize resides and then returns the value of valueToReturn in the row.

For more information about built-in aggregate functions, see Aggregate functions.

July 29, 2020 (UTC+8): The default data type edition for a new project is changed from the MaxCompute V1.0 data type edition to the MaxCompute V2.0 data type edition

If you create a MaxCompute project in the DataWorks console, the default data type edition of the project is changed from the MaxCompute V1.0 data type edition to the MaxCompute V2.0 data type edition. This change applies to all regions from July 29, 2020 to August 6, 2020 at the international site (alibabacloud.com) where MaxCompute is available. If you have created projects by using your Alibaba Cloud account, make sure that the data type edition you select for a new project is the same as that of the existing projects. Incompatibility issues may occur if projects of different data type editions interact with each other.

MaxCompute provides the following data type editions: MaxCompute V1.0 data type edition, MaxCompute V2.0 data type edition, and Hive-compatible data type edition. These editions are different in definitions and usage. MaxCompute provides three attributes to configure data type editions. You can configure these attributes to enable the edition that you want to use. For more information about data type editions, see Data type editions.

Note

This feature has no negative impact on the data type editions of existing MaxCompute projects. You can change the data type editions of existing MaxCompute projects. For more information, see the "Change the data type edition of a project" section in Data type editions.

June 29, 2020 (UTC+8): Users can choose data type editions for new projects

This feature applies to all regions from June 29, 2020 to July 15, 2020 at the international site (alibabacloud.com) where MaxCompute is available. After the feature is available, you must select initial data type editions for new MaxCompute projects.

MaxCompute provides the following data type editions: MaxCompute V1.0 data type edition, MaxCompute V2.0 data type edition, and Hive-compatible data type edition. These editions are different in definitions and usage. MaxCompute provides three attributes to configure data type editions. You can configure these attributes to enable the edition that you want to use. For more information about data type editions, see Data type editions.

Note

This feature has no negative impact on the data type editions of existing MaxCompute projects. You can change the data type editions of existing MaxCompute projects. For more information, see the "Change the data type edition of a project" section in Data type editions.

March 15, 2020 (UTC+8): The storage price of MaxCompute is reduced

From March 15, 2020, the storage price of MaxCompute is reduced. For more information, see Storage pricing (pay-as-you-go). The price is reduced based on the following rules:

  • The number of pricing tiers is reduced from five to three to simplify storage billing.

  • The unit price of each new tier is reduced to lower the overall storage price.

The following table describes the tiered pricing method that was used before March 15, 2020.

Volume of stored data

Tiered unit price

Fixed price

0 < Data volume ≤ 1 GB

N/A.

0.00 USD per GB-day

1 GB < Data volume ≤ 100 GB

0.0028 USD per GB-day

N/A

100 GB < Data volume ≤ 1 TB

0.0014 USD/GB/day

N/A

1 TB < Data volume ≤ 10 TB

0.0013 USD/GB/day

N/A

10 TB < Data volume ≤ 100 TB

0.0011 USD/GB/day

N/A

Data volume > 100 TB

0.0009 USD/GB/day

N/A

The following table describes the tiered pricing method that is used since March 15, 2020.

Volume of stored data

Tiered unit price

Fixed price

0 < Data volume ≤ 1 GB

N/A.

0.00 USD per day

1 GB < Data volume ≤ 10 TB

0.0011 USD/GB/day

N/A

10 TB < Data volume ≤ 100 TB

0.0009 USD/GB/day

N/A

Data volume > 100 TB

0.0006 USD/GB/day

N/A

The storage billing method remains unchanged. For more information, see Storage pricing (pay-as-you-go).

  • You are charged for daily stored data, including tables and resources, in MaxCompute by tier based on the data volume.

  • MaxCompute records the volume of data that is stored in each project on an hourly basis and then calculates the average volume of stored data for each day. The storage fee is equal to the average volume of stored data multiplied by the unit price. MaxCompute calculates the average volume of stored data in each project in a day. Therefore, if more data is stored in a specific project, the storage fee is low.

For example, the daily average data volume of a project is 1 PB, and the following formula is used to calculate the daily fee based on the original tiered pricing method:

(100 - 1) GB × USD 0.0028 per GB per day 
+ (1024 - 100) GB × USD 0.0014 per GB per day 
+ (10240 - 1024) GB × USD 0.0013 per GB per day 
+ (102400 - 10240) GB × USD 0.0011 per GB per day 
+ (10240 × 10240 - 102400) GB × USD 0.0009 per GB per day
= USD 966.486 per day

The following formula is used to calculate the daily fee based on the new tiered pricing method:

(10240 - 1) GB × USD 0.0011 per GB per day
+ (102400 - 10240) GB × USD 0.0009 per GB per day 
+ (10240 × 10240 - 102400) GB × USD 0.0006 per GB per day
= USD 661.9125 per day

February 24, 2020 (UTC+8): The SQL engine is updated for better compatibility

The following table describes the schedule for updating the SQL engine. If any change occurs, the new schedule prevails.

Sequence

Region

Upgrade duration

1

Indonesia (Jakarta), UK (London), and India (Mumbai)

February 24, 2020

2

UAE (Dubai), US (Virginia), China North 2 Ali Gov, and China (Hong Kong)

February 26, 2020

3

Malaysia (Kuala Lumpur), Japan (Tokyo), and Germany (Frankfurt)

March 2, 2020

4

US (Silicon Valley), Singapore, and Australia (Sydney)

March 4, 2020

  • Changes to the execution rule of the GET_IDCARD_AGE function

    • If the difference between the current year and the year of birth is greater than 100, NULL is returned, based on the original execution rule of the GET_IDCARD_AGE function. After the new rule is applied, the difference between the current year and the year of birth is returned even if the difference exceeds 100. For example, the execution result of get_idcard_age('110101190001011009') is NULL before the change and 120 after the change.

    • If you want to apply the original execution rule to a query statement after the change, you must find the query statement and modify this statement based on your business requirements. For example, you can add the IF function or CASE WHEN expression for processing the return result of the get_idcard_age function to the query statement.

      Query statement before the change

      Query statement after the change

      GET_IDCARD_AGE(idcardno)

      if(GET_IDCARD_AGE(idcardno) > 100, NULL, GET_IDCARD_AGE(idcardno))

      GET_IDCARD_AGE(idcardno)

      CASE WHEN GET_IDCARD_AGE(idcardno) > 100 THEN NULL ELSE GET_IDCARD_AGE(idcardno) END

  • Changes to the execution rule of the CONCAT_WS function

    • Before the change, if the CONCAT_WS function that is used in a query does not support Hive and has three or more parameters, including at least one parameter of the ARRAY type, the array items do not appear in the final result. For example, the expected result of the concat_ws(',', array('a'), array('b', 'c')) function is "a,b,c", but the actual result is ",,,".

    • After the change, the parameters of the STRING and ARRAY types can coexist in the CONCAT_WS function without requiring you to enable the Hive-compatible data type edition. For example, the returned result of the concat_ws(',', array('a'), array('b', 'c')) function is "a,b,c".

  • Changes to the execution rule of the Like%% function when the input value is an empty string

    Before the change, if the input value for the character matching function LIKE is an empty string and its pattern is %%, False is returned. After the change, True is returned.

    -- Create a table and insert an empty string into the table. 
    create table if not exists table_test (a string) lifecycle 3;
    insert into table table_test values ('');
    
    select a like '%%' from table_test;
    
    -- The following result is returned before the change. 
    +------+
    | _c0 |
    +------+
    | false |
    +------+
    
    -- The following result is returned after the change. 
    +------+
    | _c0 |
    +------+
    | true |
    +------+

December 25, 2019 (UTC+8): MaxCompute is compatible with open source geospatial UDFs

MaxCompute is compatible with open source geospatial UDFs, which are implemented by Esri for Apache Hive. You can register open source geospatial UDFs with MaxCompute so that the functions can be called the same as open source Hive UDFs. For more information, see Geospatial UDFs.

October 11, 2019 (UTC+8): New features of MaxCompute SQL are introduced

  • Support for specifying the priorities of the JOIN operations or SET operations, including UNION, INTERSECT, and EXCEPT by using parentheses ()

    SELECT * FROM src JOIN (src2 JOIN src3 on xxx) ON yyy; 
    SELECT * FROM src UNION ALL (SELECT * FROM src2 UNION ALL SELECT * FROM src3);

    For more information, see JOIN and INTERSECT, UNION, and EXCEPT.

  • Support for odps.sql.orderby.position.alias and odps.sql.groupby.position.alias

    If the two flags are enabled, integer constants in the ORDER BY and GROUP BY clauses of the SELECT statements are processed as column IDs.

    The columns in the src table can be sorted based on keys or values.
    SELECT * FROM src ORDER BY 1;
    -- The preceding statement is equivalent to the following statement:
    SELECT * FROM src ORDER BY key;

    For more information, see SELECT syntax.

  • Support for new built-in functions

    • STRING JSON_TUPLE(STRING json,STRING key1,STRING key2,…): converts a JSON string into a tuple based on a group of keys. The JSON_TUPLE() function supports multi-level nesting. It can parse JSON data that contains Chinese characters or nested arrays. However, the JSON_TUPLE() function can parse the JSON string only once after you enter multiple keys at a time. JSON_TUPLE() is more efficient than GET_JSON_OBJECT(). For more information, see String functions.

    • INT EXTRACT(datepart from timestamp): extracts a part specified by the datepart parameter from a date value. The value of datepart can be YEAR, MONTH, or DAY. The value of timestamp is a date value of the TIMESTAMP type. For more information, see Date functions.

  • Support for the default values of columns in a table

    The DEFAULT VALUE clause allows you to specify a default value for a column when you create a table. If you do not specify a value for the column in an INSERT statement, this default value is inserted. Example:

    CREATE TABLE t (a bigint default 10, b bigint);
    INSERT INTO TABLE t(b) SELECT 100; 
    -- The preceding statement is equivalent to the following statement:
    INSERT INTO TABLE t(a, b) SELECT 10, 100;
  • Support for a NATURAL JOIN operation

    A NATURAL JOIN operation is a process in which two tables are joined based on their common columns. Common columns are columns that have the same name in the two tables. MaxCompute supports OUTER NATURAL JOIN. You can use the USING clause so that the JOIN operation returns common columns only once. Example:

    -- To join the src table that contains the key1, key2, a1, and a2 columns and the src2 table that contains the key1, key2, b1, and b2 columns, you can execute the following statement:
    SELECT * FROM src NATURAL JOIN src2;
    -- Both the src and src2 tables contain the key1 and key2 fields. In this case, the preceding statement is equivalent to the following statement:
    SELECT src.key1 as key1, src.key2 as key2, src.a1, src.a2, src2.b1, src2.b2 FROM src INNER JOIN src2 ON src.key1 = src2.key1 AND src.key2 = src2.key2;

    For more information, see JOIN.

  • Support for simultaneous execution of the OFFSET and ORDER BY LIMIT clauses

    The OFFSET clause can be used together with the ORDER BY LIMIT clause to skip the rows specified by OFFSET. For example, execute the following statement to sort the rows of the src table in ascending order by key, and return the 11th to 20th rows. OFFSET 10 indicates that the first 10 rows are skipped. LIMIT 10 indicates that a maximum of 10 rows can be returned.

    SELECT * FROM src ORDER BY key LIMIT 10 OFFSET 10;

    For more information, see SELECT syntax.

  • Support for built-in operators

    • The IS DISTINCT FROM operator is supported. a is distinct from b is equivalent to not(a <=> b), and a is not distinct from b is equivalent to a <=> b .

    • The || operator is supported to concatenate strings. For example, a || b || c is equivalent to concat(a, b, c).

    For more information, see Operators.

  • Support for partition merging

    In MaxCompute, a maximum of 60,000 partitions can be created in a table. If excessive partitions exist, you can merge partitions to archive data in a data warehouse. This reduces the number of partitions. If you trigger partition merging, MaxCompute merges multiple partitions in the same table into one partition, migrates their data to the merged partition, and then drops the previous partitions. The following syntax is used to merge partitions. For more information, see Partition and column operations.

    ALTER TABLE <tableName> MERGE [IF EXISTS] PARTITION(<predicate>) [, PARTITION(<predicate2>) ...] OVERWRITE PARTITION(<fullPartitionSpec>) ;
  • Add/Drop Partitions

    MaxCompute allows you to add or drop multiple partitions at a time by using statements. Syntax for the statements:

    ALTER TABLE t ADD [IF NOT EXISTS] PARTITION (p = '1') PARTITION (p = '2');
    ALTER TABLE t DROP [IF EXISTS]  PARTITION (p = '1'), PARTITION (p = '2');
    -- Note that no commas (,) exist between partitions in the ADD clause, whereas commas (,) are used to separate partitions in the DROP clause.

August 29, 2019 (UTC+8): A custom storage handler for an external table is used to update the Outputer interface in some regions

On August 29, 2019, MaxCompute is updated. The update may fail if you use a custom storage handler for an external table to update the Outputer interface and the column data is obtained by column name instead of array subscript.

Upgrade time: 14:00 to 23:00 on August 29, 2019 (UTC+8)

Regions: US (Silicon Valley) and Singapore

August 21, 2019 (UTC+8): A custom storage handler for an external table is used to update the Outputer interface in some regions

On August 21, 2019, MaxCompute is updated. The update may fail if you use a custom storage handler for an external table to update the Outputer interface and the column data is obtained by column name instead of array subscript.

Upgrade time: 14:00 to 23:00 on August 21, 2019 (UTC+8)

Regions: Japan (Tokyo), Germany (Frankfurt), China (Hong Kong), and Australia (Sydney)

Impact: In Outputer.output(Record record), the input record is generated by the last operator of Outputer. Column names are not fixed.

For example, the column name that is generated by the some_function(column_a) expression is a temporary column name. We recommend that you use record.get(index) instead of record.get(Column name) to obtain the content of a column. To obtain column names from a table in Outputer, call DataAttributes.getFullTableColumns().

July 24, 2019 (UTC+8): Spark on MaxCompute is supported

Regions: China (Hangzhou), China (Beijing), China (Shenzhen), US (Silicon Valley), China (Hong Kong), Germany (Frankfurt), Singapore, and India (Mumbai)

March 26, 2019 (UTC+8): MaxCompute SQL is updated

  • The GROUPING SETS clause can be used with both the CUBE and ROLLUP subclauses to aggregate and analyze data of multiple dimensions. For example, you must aggregate Column a, aggregate Column b, or aggregate both Column a and Column b. For more information, see GROUPING SETS.

  • The INTERSECT, MINUS, and EXCEPT clauses are supported. For more information, see INTERSECT, UNION, and EXCEPT.

  • If MaxCompute reads files in the ORC or Parquet format in OSS by using external tables, MaxCompute prunes the columns in the files to reduce I/O, save resources, and lower overall computing costs.

  • Systems that run in the Java UDX framework are enhanced to support writable parameters. For more information, see Java UDFs.

Optimized SQL performance

  • DynamicDAG: a required mechanism for dynamic optimization. DynamicDAG delays the optimization of resource allocation or algorithm selection and triggers it at runtime to improve optimizations and reduce the probability of generating low-performance execution plans.

  • ShuffleRemove optimization: optimization for shuffles. MaxCompute supports ShuffleRemove for right tables that have duplicate keys during the execution of the INNER JOIN clause.

March 1, 2019 (UTC+8): MaxCompute SQL jobs that process data in external tables begin to incur charges

As of March 1, 2019, all MaxCompute SQL jobs that are used to process data in OSS and Tablestore tables begin to incur charges.

Billing standard for SQL computing that involves external tables:

Fee of an SQL job that involves external tables = Amount of input data × Unit price for SQL computing that involves external tables

The unit price for SQL computing that involves external tables is USD 0.0044 per GB. All fees are charged the next day, and you will receive a bill in your account. For more information, see Overview.

16:00 to 20:00 on January 15, 2019 (UTC+8): The underlying metadata warehouse of MaxCompute in the China (Hong Kong) region is optimized

The underlying metadata warehouse of MaxCompute in the China (Hong Kong) region is optimized from 16:00 to 20:00 on January 15, 2019 (UTC+8). This optimization improves the performance and stability of MaxCompute. During the optimization, users in the China (Hong Kong) region may encounter job submission delays or job failures for about 1 minute. In the worst cases, applications may be unavailable for up to 30 minutes. We recommend that you do not submit jobs during optimization. Users in other regions are not affected. If you have any questions, contact MaxCompute technical support by using the enterprise DingTalk group.

December 24, 2018 (UTC+8): MaxCompute allows you to configure time zones

The default time zone of MaxCompute projects is UTC+8. The system executes time-related built-in functions and calculates fields of the DATETIME, TIMESTAMP, and DATE types based on UTC+8. As of December 24, 2018, you can configure time zones in MaxCompute for a session or project.

  • To configure a time zone for a session, execute the set odps.sql.timezone=<timezoneid>; statement along with a computing statement. Example:

    set odps.sql.timezone=Asia/Tokyo;
    select getdate();
    -- The following result is returned: 
    output:
    +------------+
    | _c0        |
    +------------+
    | 2018-10-30 23:49:50 |
    +------------+
  • To configure a time zone for a project, execute the setProject odps.sql.timezone=<timezoneid>; statement as the project owner. After the time zone of a project is configured, the time zone is used for all time computing, and the data of existing jobs is affected. Therefore, exercise caution when you perform this operation. We recommend that you perform this operation only on new projects.

Limits and usage notes:

  • SQL built-in date functions, UDFs, user-defined type (UDT), user-defined joins (UDJs), and the SELECT TRANSFORM statement allow you to configure time zones based on the time zone attribute of a project.

  • A time zone must be configured in the format such as Asia/Shanghai, which supports daylight saving time. Do not configure the time zone in the GMT+9 format.

  • If the time zone in the SDK differs from that of the project, you must configure the GMT time zone to convert the data type from DATETIME to STRING.

  • After the time zone is configured, differences exist between the real time and the output time of the related SQL statements that you execute in DataWorks. Between the years of 1900 and 1928, the time difference is 352 seconds. Before the year of 1900, the time difference is 9 seconds.

  • MaxCompute, SDK for Java, and the related client are updated to ensure that data of the DATETIME type in MaxCompute is accurate across time zones. The destination versions of SDK for Java and the related client have the -oversea suffix. The update may affect the display of data of the DATETIME type that was generated before January 1, 1928 in MaxCompute.

  • If the local time zone is not UTC+8 when you update MaxCompute, we recommend that you also update the SDK for Java and the related client. This ensures that the SQL-based computing result and data that is transferred by using Tunnel commands after January 1, 1900 are accurate and consistent. For DATETIME data that was generated before January 1, 1900, the SQL-based computing result and data that is transferred by using Tunnel commands may differ up to 343 seconds. For DATETIME data that was generated before January 1, 1928 and was uploaded before SDK for Java and the related client are updated, the time in the new version is 352 seconds earlier.

  • If you do not update the SDK for Java or the related client to versions with the -oversea suffix, the SQL-based computing result may differ from data that is transferred by using Tunnel commands. For data that was generated before January 1, 1900, the time difference is 9 seconds. For data that was generated within the period from January 1, 1900 to January 1, 1928, the time difference is 352 seconds.

    Note

    The modification of the configuration of time zones in SDK for Java or on the related client does not affect the configuration of time zones in DataWorks. Therefore, the time zones are different. You must evaluate how this may affect scheduled jobs in DataWorks. The time zone of a DataWorks server in the Japan (Tokyo) region is GMT+9, and that in the Singapore region is GMT+8.

  • If you are using a third-party client that is connected to MaxCompute by using Java Database Connectivity (JDBC), you must configure the time zone on the client to ensure that the time of the client and that of the server are consistent.

  • MapReduce supports the configuration of time zones.

  • Spark on MaxCompute supports the configuration of time zones.

    • If jobs are submitted to the MaxCompute computing cluster, the time zone of the project is automatically obtained.

    • If jobs are submitted from spark-shell, spark-sql, or pyspark in yarn-client mode, you must configure parameters in the spark-defaults.conf file of the driver and add spark.driver.extraJavaOptions -Duser.timezone=America/Los_Angeles. The timezone parameter specifies the time zone you want to use.

  • Platform for AI (PAI) supports the configuration of time zones.

  • Graph supports the configuration of time zones.