MaxCompute:Tiered storage

Key limitations

Rules cannot be configured at a more granular level:

• You cannot set an independent rule for a single non-partitioned table. It inherits the project-level rule.

• You cannot set an independent rule for a specific partition within a partitioned table. It inherits the rule of its parent partitioned table.

Rule execution

• If a table or partition meets the lifecycle criteria for both long-term storage (longterm) and low-frequency storage (lowfrequency), it is converted to longterm with priority.

• If a table or partition first meets the lowfrequency lifecycle rule, it is converted to lowfrequency. If it later meets the longterm rule, it is converted to longterm. In this case, the conversion from lowfrequency to longterm does not incur a lowfrequency access fee. For billing details, see Tiered Storage Billing.

• The platform scans for rule compliance twice a day. Therefore, there may be a delay between when a table or partition meets the conditions and when the transition occurs.

Command format

• Set a lifecycle rule at the project level

  SETPROJECT odps.table.lifecycle.config=<lifecycle_config_json_string>;

  You can also configure this in the MaxCompute console:

  1. Log in to the MaxCompute console and select a region in the upper-left corner.

  2. In the left-side navigation pane, choose Manage Configurations > Projects.

  3. On the Projects page, click Manage in the Actions column for the target project.

  4. On the Project Settings page, click the Parameter Configuration tab.

  5. In the Lifecycle Configuration section, click Edit.

  6. Configure the Last Access Configuration Policy and Last Modified Configuration Policy parameters.

     • Last Access Configuration Policy: Corresponds to the DaysAfterLastAccessGreaterThan parameter.

     • Last Modified Configuration Policy: Corresponds to the DaysAfterLastModificationGreaterThan parameter.

• Set a lifecycle rule for a partitioned table

  • Set during table creation:

    CREATE [EXTERNAL] TABLE [IF NOT EXISTS] <table_name>
     [PRIMARY KEY (<pk_col_name>, <pk_col_name2>),(<col_name> <data_type> [NOT NULL] [DEFAULT <default_value>] [comment <col_comment>], ...)]
     PARTITIONED BY (<col_name> <data_type> [comment <col_comment>], ...)
    tblproperties ('lifecycle_config' = '<lifecycle_config_json_string>')
    ;

  • Modify an existing table:

    ALTER TABLE <TABLE_NAME> SET TBLPROPERTIES ('lifecycle_config' = '<lifecycle_config_json_string>');

• View the tiered storage lifecycle configuration of a partitioned table

  SHOW CREATE TABLE <table_name>;

Parameters

The following table describes the key parameters. For information about other parameters, see Common parameters.

lifecycle_config_json_string:

• When setting a lifecycle rule at the project level, the definition is as follows.

  {
    "TierToLowFrequency": {
      "DaysAfterLastModificationGreaterThan": <days>, // Transition if days since last modification exceeds this value.
      "DaysAfterLastAccessGreaterThan": <days>, // Transition if days since last access exceeds this value.
    },
    "TierToLongterm": {
      "DaysAfterLastModificationGreaterThan": <days>,
      "DaysAfterLastAccessGreaterThan": <days>
    }
    // Each condition is optional. Multiple conditions are combined with an OR relationship.
  }

• When setting a lifecycle rule for a partitioned table, the definition is as follows.

  {
    \"TierToLowFrequency\": {
      \"DaysAfterLastModificationGreaterThan\": <days>, // Transition if days since last modification exceeds this value.
      \"DaysAfterLastAccessGreaterThan\": <days>, // Transition if days since last access exceeds this value.
    },
    \"TierToLongterm\": {
      \"DaysAfterLastModificationGreaterThan\": <days>,
      \"DaysAfterLastAccessGreaterThan\": <days>
    }
    // Each condition is optional. Multiple conditions are combined with an OR relationship.
  }

• TierToLowFrequency: Identifier for low-frequency storage.

• TierToLongterm: Identifier for long-term storage.

• DaysAfterLastModificationGreaterThan: The number of days after the last modification time (LastModifiedTime) before the data is automatically transitioned.

• DaysAfterLastAccessGreaterThan: The number of days after the last access time (LastAccessTime) before the data is automatically transitioned. If the LastAccessTime of a table or partition is empty, the following rules apply:

  • For tables or partitions created before October 1, 2023, the time is calculated by default based on the UTC+0 time zone, such as 2023.10.01 00:00:00.

  • For tables or partitions created on or after October 1, 2023, if the data has never been accessed, the calculation is based on its creation time (CreateTime).

Examples

• Example 1: Set a lifecycle rule at the project level.

  setproject odps.table.lifecycle.config={"TierToLongterm":{"DaysAfterLastAccessGreaterThan":180},"TierToLowFrequency":{"DaysAfterLastAccessGreaterThan":120}};

• Example 2: Remove a project-level lifecycle configuration.

  setproject odps.table.lifecycle.config=;

• Example 3: Set a lifecycle rule for a partitioned table.

  -- Set during table creation
  CREATE TABLE lifecycle_part_t (key string) 
  PARTITIONED BY (ds  STRING)
  tblproperties ('lifecycle_config' = '{\"TierToLowFrequency\": {\"DaysAfterLastModificationGreaterThan\": 2,\"DaysAfterLastAccessGreaterThan\": 2},\"TierToLongterm\": {\"DaysAfterLastModificationGreaterThan\": 4,\"DaysAfterLastAccessGreaterThan\": 7}}')
  ;
  -- Modify an existing table
  ALTER TABLE lifecycle_part_t SET tblproperties ('lifecycle_config'='{\"TierToLowFrequency\": {\"DaysAfterLastModificationGreaterThan\": 90,\"DaysAfterLastAccessGreaterThan\": 30},\"TierToLongterm\": {\"DaysAfterLastModificationGreaterThan\": 180,\"DaysAfterLastAccessGreaterThan\": 7}}');

• Example 4: Remove the tiered storage lifecycle configuration from a partitioned table.

  ALTER TABLE lifecycle_part_t SET tblproperties ('lifecycle_config'='{}');

Prerequisites

The tenant-level access control switch for tiered storage is enabled. To do this, follow these steps:

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Tenants .

3. On the Tenant Property tab, enable the Tenant-level Schema Syntax switch.

Note: After you enable the switch, users with only SELECT permission will no longer be able to access data in low-frequency or long-term storage. Grant the necessary permissions in advance to prevent existing jobs from failing.

Step 1: Check storage type

The StorageTier field in the table or partition properties specifies the current storage type. Only data stored in low-frequency (lowfrequency) or long-term (longterm) tiers is subject to permission constraints.

-- View table properties
DESC EXTENDED <table_name>;
-- View partition properties
DESC EXTENDED <table_name> PARTITION(<partition_spec>);

Step 2: Grant access permissions

Step 3: Verify permissions

Use the authorized account to run the following commands to check if the permissions have taken effect, and then access the target data to verify.

-- View the current user's permissions. ACL and Policy authorizations are displayed in their respective Authorization Type sections.
SHOW GRANTS;

-- Access data in low-frequency or long-term storage.
SELECT * FROM table_name;

If the authorization is effective, the query returns results. If not, an error is reported with the following message:

FAILED: ODPS-0130013:Authorization exception - Authorization failed [4009, 4019], You have no privilege 'odps:ReadLowfrequencyStorage' on {acs:odps:*:projects/<project_name>/tables/<table_name>}

Restore previous access behavior

To restore the original behavior, simply disable the tenant-level tiered storage access control switch. After disabling it, users with SELECT permission will be able to access data in all storage types. Any permissions already granted will not be affected.

Syntax

struct GET_PARTITION_META(<tableName>, <pt_col1>, <pt_col2>, ..., <pt_col_n>);

Parameters

Return value

Returns a Struct of the type struct<storagetier:string>. The struct contains a single String field that describes the storage type of the corresponding partition.