Data Lake Formation:Lifecycle management

How it works

Lifecycle rules define when a transition happens. DLF evaluates each rule based on one of four time-based triggers and transitions matching data to the target storage class.

• Scheduled rules run every night and take effect before 08:00 the next day.

• Manually triggered rules take effect immediately.

Lifecycle management applies only to structured data. For unstructured data, use lifecycle management in OSS.

Rule types

DLF supports four rule types, each using a different time reference as the trigger:

For tables without partitions, DLF uses the corresponding table-level timestamp instead.

Use cases

• Order data archival: Order tables are partitioned by date (for example, 20220101). Only the last three years of data is actively analyzed. Use the Partition Value By Time rule to automatically transition older partitions to Cold Archive.

• Inactive database archival: A database from a discontinued business line needs to be archived. Use a database-level lifecycle rule to transition all data from Standard or IA to Cold Archive.

After archiving: access and recovery

To restore archived data:

• See OSS storage classes and Convert storage classes for OSS-level restore steps.

• To restore a DLF table directly, see Restore a table.

Configure lifecycle rules based on your actual access patterns. Avoid transitioning data that you may still need to access frequently.

Limits

• Unstructured data management is not supported. Use lifecycle management in OSS instead.

• Each database or table can be associated with only one lifecycle rule at a time.

• Each lifecycle rule can be associated with a maximum of 1,000 resources.

• Table rules take priority over database rules. If a table is already bound to a database rule, the database rule is replaced for that table.

• Permissions for databases and tables are subject to DLF data permission control. Configure rules only for resources within your authorized permissions.

Create a lifecycle rule

Before you begin, ensure that you have:

• An active OSS instance. If OSS is not activated, go to the OSS console to activate it.

• The required DLF permissions for the databases or tables you want to manage.

Steps:

1. Log on to the DLF console.

2. In the left-side navigation pane, choose Lake Management > Lifecycle Management.

3. Click Create Rule and fill in the basic information:

   • Name: A unique name for the rule.

   • Description: (Optional) A brief description.

   • Catalog List: The catalog to apply the rule to.

   • Resource Type: Select Database or Table.

4. Select a rule type from the four options described in Rule types.

5. Set the transition interval — how long after the trigger condition is met before data transitions to IA, Archive, or Cold Archive.

6. Configure the execution mechanism:

   • Turn on Execution Scheduling to run the rule automatically every night. Rules take effect before 08:00 the next day.

   • Leave it off to run the rule manually from the Lifecycle Management page when needed.

7. Click Next to select the databases or tables to bind to this rule.

8. Click Add Database Resource, select the resources (search and cross-page selection are supported), then click Add.

9. Click OK to confirm the binding. If binding succeeds, the console shows the number of successfully bound resources. If any fail, the console displays the reason.

If Resource Type is set to Database, only database resources can be added. If set to Table, only table resources can be added.

Each resource can be bound to only one lifecycle rule at a time. Binding a resource to a new rule replaces its existing rule.

After adding resources, click Save to finalize.

Edit a lifecycle rule

1. Go to Lake Management > Lifecycle Management.

2. Find the rule, then click Edit in the Actions column.

View a lifecycle rule

1. Go to Lake Management > Lifecycle Management.

2. Click the rule ID to view its details, organized into three tabs:

   • Basic Information: Rule configuration, details, and execution mechanism.

   • Resource Information: Associated databases or tables.

   • Execution History: Records of both manual and scheduled executions.

Delete a lifecycle rule

1. Go to Lake Management > Lifecycle Management.

2. Find the rule, click Delete in the Actions column, then confirm by clicking Delete in the dialog.

After deletion, the rule cannot run — either manually or on a schedule. Data that was already transitioned by the rule remains in its current storage class.

Run a lifecycle rule manually

1. Go to Lake Management > Lifecycle Management.

2. Find the rule, then click Manual Execution in the Actions column.

3. Read the prompt carefully, then click OK to start the task.

View execution history

1. Go to Lake Management > Lifecycle Management.

2. Click the Execution History tab to view all past scheduled and manual execution records.

3. Click a task name to view its execution details and logs.

Restore a table

If a table has been transitioned to Archive or Cold Archive, restore it to Standard Storage before compute engines can access it.

1. In the left-side navigation pane, choose Metadata > Metadata.

2. Click the Table tab, then click the table name.

3. Click the Storage Rule tab, then click Restore Table. This transitions the storage class back to Standard Storage.

To restore cold data at the OSS level (for example, using OSS-HDFS automatic storage tiering), see:

• Convert storage classes

• How to use automatic storage tiering of OSS-HDFS