Hologres:Data mirroring

How it works

Mirroring operates at two levels: metadata and data.

Metadata mirroring starts automatically the first time you query a data lake table. Hologres triggers a background sync task that pulls the latest metadata from the source table and caches it locally. The default sync interval is one minute. Subsequent queries use the cached metadata, eliminating repeated round-trips to the external catalog.

Data mirroring must be enabled manually using ALTER EXTERNAL TABLE. Once enabled, Hologres copies the actual table data—not just metadata—into local storage and keeps it in sync with the source. Data mirroring supports:

• Paimon append-only tables and primary key tables with deletion vectors enabled

• Non-partitioned tables and partitioned tables, including specific partition subsets

• Schema evolution: adding, deleting, reordering, and renaming columns. Unsupported schema changes fall back to reading the external table directly.

• Automatic discovery and synchronization of new partitions

• Common Hologres indexes (clustering key, dictionary encoding, bitmap) on the mirrored data to reduce I/O and accelerate queries

• Querying mirrored data from secondary instances, virtual warehouse instances, and serverless resources

• Viewing the latest partition of a full mirror

• Viewing the mirror storage usage at the instance, database, and table levels

• Modifying a table's data mirroring configurations

• Disabling data mirroring for a table

When to use data mirroring

Data mirroring provides the most benefit when:

• Queries repeatedly scan the same data lake tables or partitions

• Network latency to the remote storage is high or unstable

• Query performance is bottlenecked by remote I/O, not by Hologres compute

For tables accessed only occasionally, metadata mirroring alone may be sufficient.

Prerequisites and limitations

Before you enable data mirroring, ensure that you have:

• Data Lake Formation (DLF) latest or later activated, with Apache Paimon tables created.

• A Hologres instance with the data lake acceleration service enabled (see Speed up access to OSS with DLF).

• An External Database created to map data lake table metadata to Hologres (see CREATE EXTERNAL DATABASE).

Data mirroring has the following limitations:

• External tables must use the External Database mapping method. The foreign table method is not supported.

• Only external tables whose metadata is managed by Data Lake Formation (DLF) are supported.

• Mirroring data from Apache Paimon branches is not currently supported.

Enable data mirroring

ALTER EXTERNAL TABLE <ext_db.ext_schema.ext_table> 
SET(
   -- Enable data mirroring
  data_mirroring_speed_up_enable = 'on|off', 
  [data_mirroring_partition_num='1~N|all',] |[data_mirroring_partition_list='pt1,pt2...',] 
  
  -- Assign resources to data mirroring
  [data_mirroring_guc_hg_computing_resource='[serverless | local]',]
  [data_mirroring_guc_hg_experimental_serverless_computing_required_cores='<num>',] 

  -- Index
  [data_mirroring_clustering_key='[columnName{:asc]} [,...]]',]
  [data_mirroring_dictionary_encoding_columns='[columnName [,...]]',]
  [data_mirroring_bitmap_columns='[columnName [,...]]',]
  );

ALTER EXTERNAL TABLE <ext_db.ext_schema.ext_table>
SET(
  -- Enable data mirroring
  data_mirroring_speed_up_enable = 'on',

  -- Specify the resources for the mirror build process
  data_mirroring_guc_hg_computing_resource = 'local',

  -- Set indexes
  data_mirroring_clustering_key = 'c_int:asc,c_char:desc',
  data_mirroring_dictionary_encoding_columns = 'c_int:on,c_char:auto',
  data_mirroring_bitmap_columns = 'c_int:on'
);

ALTER EXTERNAL TABLE <ext_db.ext_schema.ext_table>
SET(
  data_mirroring_speed_up_enable = 'on',
  data_mirroring_partition_num = '7',

  -- Use Serverless resources for the mirror build process
  data_mirroring_guc_hg_computing_resource = 'serverless',
  data_mirroring_guc_hg_experimental_serverless_computing_required_cores = '16',

  data_mirroring_clustering_key = 'c_int:asc,c_char:desc',
  data_mirroring_dictionary_encoding_columns = 'c_int:on,c_char:auto',
  data_mirroring_bitmap_columns = 'c_int:on'
);

ALTER EXTERNAL TABLE <ext_db.ext_schema.ext_table>
SET(
  data_mirroring_speed_up_enable = 'on',
  data_mirroring_partition_num = 'all',

  data_mirroring_guc_hg_computing_resource = 'local',

  data_mirroring_clustering_key = 'c_int:asc,c_char:desc',
  data_mirroring_dictionary_encoding_columns = 'c_int:on,c_char:auto',
  data_mirroring_bitmap_columns = 'c_int:on'
);

ALTER EXTERNAL TABLE <ext_db.ext_schema.ext_table>
SET(
  data_mirroring_speed_up_enable = 'on',
  data_mirroring_partition_list = 'pt1,pt2',

  data_mirroring_guc_hg_computing_resource = 'local',

  data_mirroring_clustering_key = 'c_int:asc,c_char:desc',
  data_mirroring_dictionary_encoding_columns = 'c_int:on,c_char:auto',
  data_mirroring_bitmap_columns = 'c_int:on'
);

ALTER EXTERNAL TABLE <ext_db.ext_schema.ext_table>
SET(
  data_mirroring_speed_up_enable = 'on',
  data_mirroring_partition_num = '7',
  data_mirroring_partition_list = 'pt1',

  data_mirroring_guc_hg_computing_resource = 'local',

  data_mirroring_clustering_key = 'c_int:asc,c_char:desc',
  data_mirroring_dictionary_encoding_columns = 'c_int:on,c_char:auto',
  data_mirroring_bitmap_columns = 'c_int:on'
);

Monitor mirroring

Use the following tools depending on what you want to check.

SELECT * FROM hologres.hg_datalake_get_mirror_config();

SELECT * FROM hologres.hg_datalake_get_mirror_status();

SELECT *
FROM hologres.hg_user_datalake_mirror_cron_tasks
WHERE command::jsonb->>'external_db_name' = '<external_database_name>'
ORDER BY start_time DESC;

EXPLAIN ANALYZE SELECT <columns> FROM <holo_ext_db.ext_schema.ext_table>;

Meta mirror table count: use 1 miss 0.
Data mirror file count: use 12 miss 0.

Modify the mirror configuration

To update the mirror configuration—for example, to change the number of partitions—run ALTER EXTERNAL TABLE again with the updated parameters:

ALTER EXTERNAL TABLE <ext_db.ext_schema.ext_table>
SET(
  data_mirroring_speed_up_enable = 'on',
  data_mirroring_partition_num = '12'
);

Disable data mirroring

To disable mirroring on a table, set data_mirroring_speed_up_enable to off:

ALTER EXTERNAL TABLE <holo_ext_db.ext_schema.ext_table>
SET(
  data_mirroring_speed_up_enable = 'off'
);

After mirroring is disabled, Hologres stops syncing data and queries access the data lake directly. The locally mirrored data files are asynchronously deleted within 30 minutes.

What's next

• Set table properties and indexes — learn more about the index types supported on mirrored data

• OSS data lake acceleration — overview of the data lake acceleration service

• CREATE EXTERNAL DATABASE — set up the External Database required for data mirroring