Data Lake Formation:Storage optimization

Edit the policy pattern as needed:

Dynamic resource pattern (Recommended)

The system automatically scales computing resources based on the real-time load. This pattern does not require manual capacity planning and is suitable for scenarios with fluctuating traffic.
Three configuration preferences are supported:

• Balanced resource and latency: Strikes a balance between merge speed and resource consumption (default).

• Latency first: Allocates more resources to complete merges faster and reduce data visibility latency.

• Resource first: Limits resource usage to reduce computing costs, which may result in longer merge times.

  Dynamic resource allocation and scaling policy

  In the dynamic resource pattern, the system automatically allocates computing resources for primary key tables based on real-time load and data features. The amount of allocated resources is determined by the following core factors and triggers automatic scaling when necessary.

  Write throughput
  Write traffic is the main driver for resource allocation. Higher write traffic results in more computing resources being allocated to ensure stable data ingestion.
  
  

  Active concurrency
  The number of active partitions and buckets directly determines the resource requirements. The more active partitions and buckets there are, the more parallel processing resources the system allocates.
  
  

  Data volume for advanced features
  When a table has deletion vectors or the lookup changelog producer enabled, the computational complexity increases. In this case, the larger the total file size in active partitions, the more resources the system allocates.
  
  

  Data row features
  Extreme sizes of single rows increase processing overhead. Whether a single row is too small (contains many null values) or too large (contains very long string fields), the system increases resource allocation to maintain processing performance.
  
  

  Resource optimization policy for small tables

  For small-scale data tables, the system uses a sharing mechanism to reduce resource overhead:

  When a table is configured for adaptive bucketing (bucket = -2) and "Latency first" mode is not enabled, the system merges the optimization tasks of multiple small tables into a single job. This multi-table sharing mechanism significantly reduces overall resource usage.

  Resource monitoring and troubleshooting

  To analyze resource consumption, go to Data Catalog > Resource Overview > Resource Request Metric Details and view the CU * Hours Top Tables list.

  Note

  If a single table shows abnormally high resource usage, it is often because the partition granularity is too fine. A fine-grained partition policy can cause a surge in the number of concurrently active partitions, forcing the system to allocate many resources. Check and optimize the partition policy.

Fixed resource pattern

Manually specify the amount of computing resources for compaction. This pattern is suitable for scenarios with stable traffic or strict cost control requirements.

• Configuration requirements: The computing unit (CU) configuration must be at least 2 CUs.

• Parameter settings: You can customize the compaction trigger interval and the number of buckets.

View execution status

You can view the optimization execution status of the current table and configure custom alert subscriptions in Cloud Monitor. For more information about metrics and configuration steps, see Lakehouse table optimization monitoring.

View Rescale records

This section records historical events of bucket rescaling for a data table or specific partitions. It reflects changes in the underlying physical storage structure of the table. The rescale mechanism is mainly used to address performance issues caused by changes in data volume. You can use the Rescale records to determine if a table is not entering the compaction process because it is currently being rescaled.

View execution history

You can view the execution history of small file merges for the current table. This history shows how the system processes fragmented files to optimize read performance and storage space. Use these records to:

1. Confirm task execution: Ensure that background merge tasks are running correctly to prevent an infinite buildup of small files.

2. Evaluate compression efficiency: Compare the number of files and their sizes before and after merging to determine whether the current compaction strategy is appropriate.

Configure partition cleanup rules

1. You can click the toggle on the right side of Enable Partition Cleanup to enable partition cleanup.

2. After you enable partition cleanup, configure the following rules as needed. Click Save to complete the configuration.

   You can complete the configuration by setting the corresponding table option key-value pairs.

   Configuration item	Description
   Expiration Policy (partition.expiration-strategy)	You can select one of the following expiration policies: Last access time (access-time): Determines expiration based on the last access time of the partition data. Partition value (values-time): You can configure the partition timestamp format and partition field pattern. Timestamp format (partition.timestamp-formatter): You can configure formats such as yyyy-MM-dd, yyyyMMdd, dd/MM/yyyy, and dd.MM.yyyy. Timestamp pattern (partition.timestamp-pattern): By default, the first partition field is used. You can configure patterns such as $dt or $year-$month-$day. Last update time (update-time): Determines expiration based on the last update time of the partition data at the finest granularity.
   Partition Retention Period (partition.expiration-time)	Unit: days. You can configure a value such as 30d. The maximum value is 999,999 days. The start time for the retention period is determined by the selected expiration policy.

3. (Optional) After saving, you can also click Edit next to Rule Configuration to make changes.

View partition cleanup results

Click View Partitions to view the partition list for the current table. The list includes the partition name, number of rows (physical), number of referenced files, total file size, creator, storage class, last updater, creation time, last update time, and operations.

Configure storage tiering