Previously, compaction consumed dedicated instance resources, impacting other requests. Starting with V3.1.11, data import tasks using serverless resources can handle compaction. This approach avoids using dedicated computing resources for compaction, thus protecting the performance of other operations.
How to use
-- Enable at the session level
SET hg_serverless_computing_run_compaction_before_commit_bulk_load = on;
-- Enable at the user level
ALTER USER "<role_name>" IN DATABASE <db_name> SET hg_serverless_computing_run_compaction_before_commit_bulk_load = on;
-- Enable at the DB level
ALTER DATABASE <db_name> SET hg_serverless_computing_run_compaction_before_commit_bulk_load = on;Limitations:
This option slightly increases data import time but significantly enhances instance stability.
Concurrent compaction is limited to new data files. Cleanup of outdated data during compaction will still use dedicated instance resources.
Concurrent compaction applies only to data imported into hot storage (Standard tier), not cold storage (Infrequent Access tier).
Usage examples
Use the public dataset import feature on HoloWeb to import the TPC-H 100 GB dataset.
Disable concurrent compaction at the database level and run a data import task.
ALTER DATABASE <db_name> SET hg_serverless_computing_run_compaction_before_commit_bulk_load = off;Enable concurrent compaction at the database level and run the data import task again.
ALTER DATABASE <db_name> SET hg_serverless_computing_run_compaction_before_commit_bulk_load = on;
After both import tasks complete, check the instance's monitoring metrics. The following illustrations show that the import tasks use serverless resources as expected, while the compaction task consumes no dedicated resources from the instance's virtual warehouse.
CPU utilization of the virtual warehouse
QPS of import tasks
Serverless resource utilization
