MaxCompute:Manage table lifecycle

Lifecycle

MaxCompute automatically reclaims tables or partitions whose data remains unchanged beyond the configured lifecycle. The system scans for expired data daily.

• If a non-partitioned table's LastModifiedTime exceeds its lifecycle, MaxCompute reclaims the data and deletes the table.

• For partitioned tables, MaxCompute reclaims individual partitions based on their LastModifiedTime. The table remains after its last partition is reclaimed.

  To automatically delete the table after its last partition is reclaimed, use one of these methods:

  Note

  Table-level settings take precedence over project-level settings.

  • Project-level settings

    -- Automatically delete the table after the last partition of the table is reclaimed.
    setproject odps.table.lifecycle.deletemeta.on.expiration=true;
    
    -- Retain the table by default after the last partition is reclaimed.
    setproject odps.table.lifecycle.deletemeta.on.expiration=false;

  • Table-level settings

    --- Create a table and configure a lifecycle-related parameter to allow the table to be automatically deleted after the last partition of the table is reclaimed.
    CREATE TABLE <Table_Name>(id int, name string)
         partitioned BY (ds string)
         tblproperties ('lifecycle.deletemeta'='true')
         lifecycle  1;
    --- If a table exists, configure a lifecycle-related parameter to allow the table to be automatically deleted after the last partition of the table is reclaimed.
    ALTER TABLE <table_name> SET tblproperties('lifecycle.deletemeta'='true');
    
    -- Retain the table by default after the last partition is reclaimed.
    ALTER TABLE <table_name> SET tblproperties('lifecycle.deletemeta'='false');
    

Usage notes

• Task scheduling dependencies do not prevent data reclamation after the lifecycle expires.

• Lifecycle applies to the entire partitioned table. You cannot configure lifecycle at the partition level.

• Reclaimed data is retained in the recycle bin for one day by default, then permanently deleted and cannot be recovered.

• Reclamation may not occur immediately after lifecycle expiration. Timing depends on system availability.

• Data writes, updates, or deletes update LastModifiedTime. Modifying the lifecycle value does not.

Configure table lifecycle

Set lifecycle during table creation

• Syntax

  -- Create a table. 
   CREATE [external] TABLE [if not exists] <table_name>
   [(<col_name> <data_type> [default <default_value>] [comment <col_comment>], ...)]
   [comment <table_comment>]
   [partitioned BY (<col_name> <data_type> [comment <col_comment>], ...)]
   -- Configure the shuffle and sort properties of a clustered table that you want to create. 
   [clustered BY | range clustered BY (<col_name> [, <col_name>, ...]) [sorted BY (<col_name> [ASC | DESC] [, <col_name> [ASC | DESC] ...])] INTO <number_of_buckets> buckets] 
   -- Used only for external tables. 
   [stored by StorageHandler] 
   -- Used only for external tables. 
   [with serdeproperties (options)] 
   -- Used only for external tables. 
   [location <osslocation>] 
   -- Configure the lifecycle for the table.
   lifecycle <days>;
  
  -- Create a table based on an existing table and replicate the schema of the existing table. You can execute the CREATE TABLE statement to create a table based on an external table or an existing table of an external project that is used to implement the data lakehouse solution. 
  create table [if not exists] <table_name> like <existing_table_name> [lifecycle <days>];

• Parameters

  • table_name: required. The table name.

  • days: required. Lifecycle duration as a positive integer. Unit: days.

    Note

    This section covers only lifecycle-related parameters. For more information about the parameters for creating a table, see Create a table.

• Example

  -- Create the test_lifecycle table with the lifecycle of 100 days. 
  CREATE TABLE test_lifecycle (key string) lifecycle 100;

Configure or modify lifecycle of an existing table

Run the following statement to modify the lifecycle of an existing table.

• Syntax

  ALTER TABLE <Table_Name> SET lifecycle <days>;

• Parameters

  • table_name: required. The table name.

  • days: required. New lifecycle duration as a positive integer. Unit: days.

• Example

  -- Change the lifecycle of the test_lifecycle table to 50 days. 
  ALTER TABLE test_lifecycle set lifecycle 50;

Disable or restore table lifecycle

• Syntax

  ALTER TABLE <table_name> [<pt_spec>] {enable|disable} lifecycle;

• Parameters

  • table_name: required. The table name.

  • pt_spec: optional. The target partition. For multi-level partitions, specify all partition key columns in the partition_col1=col1_value1, partition_col2=col2_value1... format.

  • enable and disable: restores or disables the lifecycle for a table or partition.

    Note

    • Before restoring lifecycle, verify the configured lifecycle value to prevent unintended data reclamation.

    • Table-level disable overrides partition-level enable. If the table lifecycle is disabled, table disable lifecycle overrides pt_spec enable lifecycle.

    • Disabling lifecycle preserves settings. You can still modify lifecycle values while disabled.

• Examples

  • Example 1: Disable the lifecycle for the trans table.

    ALTER TABLE trans disable lifecycle;

  • Example 2: Disable the lifecycle for the dt='20141111' partition in the trans table.

    ALTER TABLE trans partition (dt='20141111') disable lifecycle;