MaxCompute:Storage resources - Tiered storage

Manual custom settings

For non-partitioned tables or partitions, manually set the storage class to IA storage class or long-term storage. The change takes effect immediately.

ALTER TABLE <TABLE_NAME> [PARTITION(<PARTITION_SPEC>)]
        SET <TBLPROPERTIES|PARTITIONPROPERTIES>("storagetier"="standard|lowfrequency|longterm");

Automatic configuration using lifecycle rules

You can configure tiered storage lifecycle rules at the project or partitioned table level. These rules automatically convert storage classes based on specified conditions:

• Project-level rules:

  After you set a project-level rule, it becomes the default rule for all non-partitioned tables in the project and for all partitions in partitioned tables that do not have their own rules. These tables and partitions automatically convert to the specified storage class.

• Settings at the partitioned table level:

  After you set a rule for a specific partitioned table, it takes precedence over the project-level rule. All partitions in the table that meet the rule conditions automatically convert to the specified storage class.

GET_PARTITION_META

struct GET_PARTITION_META(<tableName>, <pt_col1>, <pt_col2>, ..., <pt_col_n>);

Notes

• When adding a row-level rule to a table, consider access patterns of users other than those being controlled. If other users have already accessed the table, set explicit rules for them to avoid unexpected data filtering due to unintended access restrictions. For more information, see Row-level access control.

• By default, MaxCompute tables can be accessed through SQL and external engines (such as Spark and Flink). However, only the MaxCompute SQL engine supports the GET_PARTITION_META function. If you use GET_PARTITION_META in row-level access control, the table can only be read by the MaxCompute SQL engine. Other engines cannot access the table.

• To access data under row-level permissions, you must also have SELECT permission on the data.

• Partition pruning behavior varies depending on the filter conditions used with GET_PARTITION_META.

  • If the filter condition involves only partition fields—for example, allowing access to Standard storage where the first-level partition value is 2024:

    GET_PARTITION_META('storage_table', pt1, pt2).storagetier == 'standard') AND pt1='2024'

    When you access storage_table using SQL and the WHERE clause does not contain partition conditions, the system natively supports partition pruning to avoid a full table scan and allows read-only access only to the partitions in storage_table that meet the requirements.

  • If the filter condition includes a non-partition field combined with an AND operator—for example, allowing access to Standard storage where a>100:

    GET_PARTITION_META('storage_table', pt1, pt2).storagetier == 'standard') AND a > 100

    When you query storage_table without a partition condition in the WHERE clause, partition pruning still applies. Only partitions stored as standard are accessed.

  • If the filter condition includes a non-partition field combined with an OR operator—for example, allowing access to Standard storage or where a>100:

    get_partition_meta('storage_table', pt1, pt2).storagetier == 'standard') OR a > 100

    When you query storage_table, two scenarios apply:

    • If the WHERE clause has no partition condition, all partitions are scanned to find data where a>100.

    • If the WHERE clause includes a partition condition, only the specified partitions are scanned for data matching standard or a>100.

Examples

Create the partitioned table storage_table.

CREATE TABLE storage_table(a BIGINT, b BIGINT) PARTITIONED BY (pt1 STRING, pt2 STRING);

• Example 1:

  Create a policy named policy01 for the DEFAULT user group. This policy allows access only to Standard storage data in storage_table. All users (including the Project Owner) are denied access to IA storage class or long-term storage data and receive empty results.

  CREATE ROW ACCESS POLICY policy01 
  ON storage_table 
  TO DEFAULT 
  FILTER USING (get_partition_meta('storage_table', pt1, pt2).storagetier == 'standard');

  If the Logview Summary shows the following keyword, row-level filtering is triggered:

  WARNING:[1,15] row access policy is enabled on table yyy_hp_storagetier.storage_table01. This indicates that the row access policy was applied during the query.

• Example 2:

  Grant user user_x permission policy02 to access IA storage class and long-term storage data in storage_table.

  CREATE ROW ACCESS POLICY policy02 
  ON storage_table 
  TO USER (user_x)  -- You can also grant to a role using TO role rolename, then assign the role to user_x.
  FILTER USING (get_partition_meta('storage_table', pt1, pt2).storagetier IN ('lowfrequency','longterm'));

  Two scenarios apply:

  • If the table has the policy01 permission configured, all users (except user_x) can access the standard storage data of storage_table, but cannot access infrequent-access or long-term storage data, and any attempt to do so returns empty values. For user_x, access to the infrequent-access and long-term storage data of storage_table is allowed, but access to standard storage data is not permitted and returns empty values.

  • If policy01 is not set, all users (except user_x) cannot access any data in storage_table (empty results). Only user_x can access IA storage class and long-term storage data but cannot access Standard storage data (empty results).

• Example 3:

  Grant user user_y permission policy03 to access all data in storage_table.

  CREATE ROW ACCESS POLICY policy03 
  ON storage_table 
  TO USER (user_y)  -- You can also grant to a role using TO role rolename, then assign the role to user_y.
  FILTER USING (true);  -- Constant true allows access to all data in the table.

  Two scenarios apply:

  • If the table has been set with the policy01 permission, all users (except user_y) can access the Standard storage data of storage_table, but cannot access infrequent access or long-term storage, and any such requests return empty values. The user user_y can access data of all storage classes in storage_table.

  • If the table has not been assigned the policy01 permission, all users (except user_y) cannot access any data stored in storage_table, and such requests return empty values. Only user_y can access data of all storage classes in storage_table.