Data Lake Formation:Monitor lake table optimization

Create and manage alert rules

1. Log on to the Cloud Monitor console.

2. In the left-side navigation pane, select Alerts > Alert Rules.

3. Click Create Alert Rule and configure the parameters.

   Parameter	Description
   Product	Data Lake Formation (DLF)
   Resource Range	Select Instance. The alert rule applies to catalogs.
   Associated Resources	Click Add Instance. In the upper-left corner, select the target region. Then, select the catalogs that you want to monitor and click OK.
   Rule Description	Click Add Rule > Simple Metric or Combined Metrics to open the Configure Rule Description panel.

Cloud Monitor metrics

• compaction_latency_between_latest_compacted_and_uncompacted_commit

  • Definition: Measures the time difference between the latest compacted commit and the latest uncompacted commit.

  • Semantics: Indicates the time span of uncompacted data in the system.

  • Use cases:

    • Evaluate data catch-up progress.

    • A high value indicates that the latest compaction task is falling further behind the write progress.

  • Note: This metric measures the lag of a compaction task relative to the latest write progress (the catch-up gap), not the waiting time for a specific piece of data.

• compaction_latency_earliest_uncompacted_commit

  • Definition: Measures the latency from the commit time of the earliest uncompacted snapshot to the current system time.

  • Semantics: Indicates the lag of compaction tasks at the scheduling level. It represents the duration that the oldest batch of committed data has been waiting for compaction.

  • Use cases:

    • Monitor the responsiveness of the scheduling system.

    • Troubleshoot insufficient compaction resources.

  • Note: In high-frequency commit scenarios, this metric may be smaller than expected as it excludes the time spent writing data.

• compaction_latency_earliest_uncompacted_file

  • Definition: Measures the latency from the creation time of the earliest uncompacted data file to the current system time.

  • Semantics: Reflects the end-to-end data freshness, from the moment data is physically stored to when it is queued for compaction.

  • Use cases:

    • Scenarios that require strict data freshness monitoring.

    • Long-window stream writing or large-batch import scenarios with long checkpoint intervals.

  • Note: This metric includes upstream data write and checkpointing times. Its value is typically higher than the compaction_latency_earliest_uncompacted_commit latency and provides a more accurate measure of the age of the oldest physical data.

  You can call the GetTableCompactionInfo API operation to view the specific timestamp.

Recommended alert configurations and examples