Data Lake Formation:Storage optimization

Edit the policy mode based on your requirements:

Dynamic resource mode (recommended)

The system automatically scales compute resources based on real-time workload, eliminating manual capacity planning. Suitable for fluctuating traffic.
Three configuration preferences are available:

• Balanced: Balances compaction speed and resource consumption (default).

• Low latency: Allocates more resources for faster compaction, reducing data visibility delay.

• Low resource usage: Limits resource usage to reduce compute costs at the expense of longer compaction time.

  Dynamic resource allocation and scaling

  In dynamic resource mode, the system allocates compute resources based on real-time workload and data characteristics. The following factors drive resource allocation and automatic scaling.

  Write throughput
  Write traffic is the primary driver. Higher write traffic triggers more compute resources for stable data ingestion.
  
  

  Active concurrency
  The number of active partitions and buckets determines resource requirements. More active partitions and buckets result in more parallel processing resources.
  
  

  Data scale for advanced features
  Tables with advanced features like deletion vectors or a lookup changelog producer increase computational complexity. Larger total file sizes within active partitions require more resources.
  
  

  Data row characteristics
  Extreme row sizes increase processing overhead. Whether rows are too small (many null values) or too large (long string fields), the system increases resource allocation to maintain performance.
  
  

  Resource optimization for small tables

  For small tables, the system uses a sharing mechanism to reduce resource overhead:

  When a table uses postponed bucketing (bucket = -2) and "Prioritize Latency" mode is not enabled, the system combines optimization tasks from multiple small tables into a single job, reducing overall resource consumption.

  Resource monitoring and troubleshooting

  To analyze resource consumption, navigate to Catalogs > Resource Overview > Resource Requests and view the Top Tables by CU-Hours list.

  Note

  Abnormally high resource usage for a single table is often caused by an overly fine-grained partitioning strategy, which increases concurrently active partitions and forces excessive resource allocation. Review and optimize your partitioning strategy.

Fixed resource mode

Manually specify compute resources for compaction. Suitable for stable traffic or strict cost control.

• Configuration requirements: Minimum 2 CU.

• Parameter settings: Customize the compaction trigger interval and bucket count.

View execution status

View the optimization execution status for the current table and configure CloudMonitor alert subscriptions through Monitor lakehouse optimization.

View rescale records

Logs bucket rescaling events for a table or specific partitions, reflecting changes in the physical storage structure. Rescaling addresses performance issues from data volume changes. Use these records to check if a table is undergoing a rescale, which prevents compaction.

View execution history

View the compaction execution history for the current table. Use these records to:

1. Confirm task execution: Verify that background compaction tasks run correctly and prevent small file accumulation.

2. Evaluate compaction efficiency: Compare file count and size before and after compaction to assess strategy effectiveness.

Partition cleanup rules

1. Click the switch next to Expired Partition Cleanup to enable it.

2. Configure the following cleanup rules, then click Save.

   You can also configure these settings through table option key-value pairs.

   Parameter	Description
   Expiration Policy (partition.expiration-strategy)	You can select one of the following expiration strategies: Based on last access time (access-time): Expires partitions based on their last access time. Based on partition value (values-time): You can configure the partition timestamp format and pattern. Timestamp format (partition.timestamp-formatter): You can configure formats such as yyyy-MM-dd, yyyyMdd, dd/MM/yyyy, and dd.MM.yyyy. Timestamp pattern (partition.timestamp-pattern): By default, the first partition field is used. You can configure patterns such as $dt or $year-$month-$day. Based on last update time (update-time): Expires partitions based on their last update time.
   Partition Retention Period (partition.expiration-time)	Unit: days. Example: 30d. The maximum value is 999,999 days. The retention period starts based on the selected expiration strategy.

3. (Optional) After saving, click Cleanup Rule Settings next to Modify to modify settings.

Partition cleanup results

Click View Partitions to view the partition list, including partition name, row count, referenced files, file size, creator, storage class, last modified by, timestamps, and actions.

Storage tiering