Compaction backlogs on Paimon primary key tables can silently degrade query performance before they become visible. Data Lake Formation (DLF) integrates with Cloud Monitor to surface three compaction latency metrics so you can set alert rules and catch backlogs early.
How compaction monitoring works
DLF runs compaction automatically at the scheduling layer. After data is committed, it waits in uncompacted snapshots until the scheduler picks it up. The three metrics described below measure this lag from different angles — the scheduling queue depth, the physical file age, and the gap between the latest compaction and the latest write. Choose the metric that matches your workload, or combine them for broader coverage.
These metrics apply only to Paimon primary key tables with storage optimization enabled (write-only=true). This option is on by default in the DLF fully managed service. When no data is pending compaction, all three metrics return 0.
Compaction latency metrics
All three metrics are reported through Cloud Monitor and can be scoped to the catalog, database, or table dimension.
| Metric | What it measures | Best for |
|---|---|---|
compaction_latency_between_latest_compacted_and_uncompacted_commit |
Time gap between the latest compact commit and the latest non-compact commit | Evaluating whether compaction keeps pace with ongoing writes |
compaction_latency_earliest_uncompacted_commit |
How long the oldest committed-but-uncompacted snapshot has been queued | Diagnosing scheduling-layer responsiveness and resource shortages |
compaction_latency_earliest_uncompacted_file |
Physical age of the oldest uncompacted data file, from creation to now | Workloads with strict data freshness requirements |
compaction_latency_between_latest_compacted_and_uncompacted_commit
Definition: The time difference between the latest compact commit and the latest non-compact commit.
What it measures: The catchup gap — how far the most recent compaction run lags behind the most recent write. A growing value means the compaction task is not keeping pace with write throughput.
Use when: You want to evaluate whether compaction is keeping up with ongoing writes at the table level.
This metric does not measure how long any specific piece of data has been waiting. It reflects only the gap between the two most recent events.
compaction_latency_earliest_uncompacted_commit
Definition: The latency calculated from the commit time of the earliest unmerged snapshot and the current system time.
What it measures: Scheduling-layer lag — how long the oldest batch of committed data has been queued for compaction. This is the primary indicator of whether the scheduling system is responsive and has sufficient resources.
Use when:
-
Monitoring the response speed of the scheduling system
-
Troubleshooting insufficient compaction resources
In high-frequency commit scenarios, this metric may read lower than expected because it does not account for the time spent writing data before the commit.
compaction_latency_earliest_uncompacted_file
Definition: The elapsed time from the creation timestamp of the earliest uncompacted data file to the current system time.
What it measures: End-to-end data age — from the moment data is physically written to when it is still waiting for compaction. This includes upstream write time and checkpoint intervals, so its value is typically higher than compaction_latency_earliest_uncompacted_commit and more accurately reflects the physical age of the oldest data.
Use when:
-
Workloads that require strict monitoring of data freshness
-
Long-window stream writing or large-batch import scenarios with long checkpoint intervals
Prerequisites
Before you begin, make sure you have:
-
A DLF catalog with at least one Paimon primary key table
-
Storage optimization enabled (
write-only=true) on the tables you want to monitor -
Access to the Cloud Monitor console
Tip: Before configuring alert rules, verify that compaction is active on your target tables using the GetTableCompactionInfo API. This helps you distinguish a configuration problem from a data availability problem when alerts do not fire.
Create an alert rule
Each alert rule has three components: a condition (which metric and threshold), a scope (which dimension to monitor), and a trigger frequency (how many consecutive cycles before the alert fires). Decide on these before you enter the console.
-
Log on to the Cloud Monitor console.
-
In the left navigation pane, choose Alerts > Alert Rules.
-
Click Create Alert Rule and configure the following parameters.
Parameter Description Product Data Lake Formation (DLF) Resource Range Select Instances. The alert rule applies to a catalog. Associated Resources Click Add Instance. In the upper-left corner, select the destination region, then select the catalog to monitor and click OK. Rule Description Click Add Rule > Simple Metric or Combined Metrics to open the Configure Rule Description panel.
Cloud Monitor metric descriptions
-
Compaction latency_latest compact commit to latest non-compact commit
-
Definition: The time difference between the latest compact commit and the latest non-compact commit.
-
Semantics: Reflects the time span of unmerged data in the current system.
-
Scenarios:
-
Evaluate data catchup progress.
-
A larger value indicates that the latest compaction task is lagging further behind the write progress.
-
-
Note: This metric measures how far the compaction task is behind the latest write progress (the catchup gap). It does not measure the waiting time for a specific piece of data.
-
-
Compaction latency_earliest non-compact commit
-
Definition: The latency calculated from the commit time of the earliest unmerged snapshot and the current system time.
-
Semantics: Reflects the lag of the compaction task at the scheduling layer. It is the time that the oldest batch of committed data has been waiting to be merged.
-
Scenarios:
-
Monitor the response speed of the scheduling system.
-
Troubleshoot insufficient compaction resources.
-
-
Note: In high-frequency commit scenarios, this metric's value may be smaller than expected because it ignores the time taken for the data writing process.
-
-
Compaction latency_earliest non-compact file
-
Definition: The difference between the current system time and the commit time of the earliest unmerged snapshot.
-
Semantics: Reflects the end-to-end timeliness of data, from being physically written to disk to waiting for a merge.
-
Scenarios:
-
Scenarios that require strict monitoring of data freshness.
-
Long-window stream writing or large batch import scenarios where checkpoint intervals are large.
-
-
Note: This metric includes the time taken for upstream data writing and its checkpoint. It is usually greater than the "earliest non-compact commit" latency and more accurately reflects the physical age of the oldest data.
For more information, see Table merge information.
-
-
When no data is pending compaction in the system, the value of all compaction latency metrics is 0.
-
These metrics apply only to Paimon primary key tables. The storage optimization feature must be enabled by setting
write-only=true. The DLF fully managed service has this option enabled by default.
Alert configurations by scenario
Use these configurations as a starting point. Adjust thresholds to match your SLA and typical compaction pace.
Monitoring frequently updated primary key tables
For tables with high write throughput, use Combined Metrics so an alert fires if either latency metric exceeds the threshold — catching backlogs whether they originate at the scheduling layer or in physical file age.
| Config | Value |
|---|---|
| Metric Type | Combined Metrics (generates alerts if one of the conditions is met) |
| Metrics | compaction_latency_earliest_uncompacted_commit and compaction_latency_earliest_uncompacted_file |
| Condition | Alert if value is greater than a specified time in seconds |
| Select Dimension | Table |
Monitoring at the catalog or database level
For a broader view across multiple tables, use a Simple Metric on the scheduling-layer indicator.
| Config | Value |
|---|---|
| Metric Type | Simple Metric |
| Metric | compaction_latency_earliest_uncompacted_commit |
| Condition | Alert if triggered more than once |
| Select Dimension | Database or catalog (leave blank for catalog) |
Sample alert rule
The following example shows a complete configuration for monitoring a frequently updated table with a 10-minute threshold on both commit-based and file-based latency.
| Parameter | Example |
|---|---|
| Alert Rule | Compaction Latency Alert |
| Metric Type | Combined Metrics |
| Alert Level | Warning (Warn) |
| Multi-metric Alert Condition | compaction_latency_earliest_uncompacted_commit: Metric Value >= 600 s |
compaction_latency_earliest_uncompacted_file: Metric Value >= 600 s |
|
| Select Dimension | Table |
| Relationship Between Metrics | Generate alerts if one of the conditions is met |
| Alert Threshold Triggers | 1 Consecutive Cycles (1 Cycle = 1 Minutes) |
What's next
-
GetTableCompactionInfo — query the current compaction state of a table before tuning alert thresholds