Hologres:Concurrent queries with shard replicas

When to add shard replicas

The clearest signal is uneven CPU utilization across worker nodes: the instance average appears moderate, but one worker node is overloaded while another is nearly idle.

This pattern indicates that most queries hit a small set of shards on one worker node. Adding replicas places copies of those shards on underutilized worker nodes, distributing the load and increasing QPS.

After rebalancing, resource utilization evens out across all worker nodes:

Shard metadata synchronization consumes resources. Add replicas only when the root cause is confirmed to be uneven query distribution, not general CPU pressure.

Replica count and shard count

Adding replicas without reducing shards wastes resources. To keep worker node load balanced, maintain this equation:

shard_count × replica_count = recommended shard count for your instance

For recommended shard counts by instance size, see Instance management.

Query table groups and current replica counts

List all table groups in the current database:

SELECT * FROM hologres.hg_table_group_properties;

Check the replica count for a specific table group:

SELECT property_value
FROM hologres.hg_table_group_properties
WHERE tablegroup_name = '<table_group_name>'
  AND property_key = 'replica_count';

Enable shard-level replication

Set replica_count to a value greater than 1. A value of 2 is typical for most workloads.

-- Set replica count for a table group
CALL hg_set_table_group_property ('<table_group_name>', 'replica_count', '<replica_count>');

Disable shard-level replication

Set replica_count back to 1:

-- Disable shard-level replication
CALL hg_set_table_group_property ('<table_group_name>', 'replica_count', '1');

Verify replica distribution

After enabling replication, confirm that each replica has been assigned to a worker node:

SELECT * FROM hologres.hg_worker_info;

Before a shard's metadata loads onto a worker node, the worker_id field for that worker may appear empty.

The result shows which shards are loaded on which worker nodes. In the example below, Shard 0 and Shard 1 of the olap_replica_2 table group are distributed across the 7tn8k and 9c8sl worker nodes:

Choose a routing strategy

Select the combination that matches your goal:

hg_experimental_query_replica_mode

Controls which replicas receive queries.

-- Session-level
SET hg_experimental_query_replica_mode = leader_follower;

-- Database-level
ALTER DATABASE <database_name>
SET hg_experimental_query_replica_mode = leader_follower;

hg_experimental_query_replica_leader_weight

Sets the traffic weight of the leader shard relative to each follower shard. Each follower always has a fixed weight of 100. Applies when hg_experimental_query_replica_mode = leader_follower.

Example: With replica_count = 4 (1 leader + 3 followers) and the default leader weight of 100, all four replicas have equal weight (25% each). Set the leader weight lower to send more traffic to followers, or higher to concentrate traffic on the leader.

-- Session-level
SET hg_experimental_query_replica_leader_weight = 100;

-- Database-level
ALTER DATABASE <database_name>
SET hg_experimental_query_replica_leader_weight = 100;

hg_experimental_query_replica_fixed_plan_ha_mode

Applies only to point queries processed with fixed plans. Controls retry behavior when a shard is unavailable.

-- Session-level
SET hg_experimental_query_replica_fixed_plan_ha_mode = any;

-- Database-level
ALTER DATABASE <database_name>
SET hg_experimental_query_replica_fixed_plan_ha_mode = any;

hg_experimental_query_replica_fixed_plan_first_query_timeout_ms

The timeout threshold in milliseconds for the first attempt of a point query in HA mode. If no result is returned within this window, the system retries on another available shard replica. Applies only to point queries processed with fixed plans.

-- Session-level
SET hg_experimental_query_replica_fixed_plan_first_query_timeout_ms = 60;

-- Database-level
ALTER DATABASE <database_name>
SET hg_experimental_query_replica_fixed_plan_first_query_timeout_ms = 60;

High throughput: distribute hot-data queries across worker nodes

Scenario: CPU utilization is uneven—one worker node is overloaded while others are idle—because most queries target a small number of shards.

Configuration:

1. Set replica_count to 2 for the affected table group:

   -- Set replica count to 2 for the tg_replica table group
   CALL hg_set_table_group_property ('tg_replica', 'replica_count', '2');

   The default routing settings apply automatically:

   • hg_experimental_query_replica_mode = leader_follower

   • hg_experimental_query_replica_leader_weight = 100

   Queries are now distributed evenly across all replicas, increasing QPS even for hot-data workloads.

2. Verify that replicas are loaded on each worker node:

   SELECT * FROM hologres.hg_worker_info WHERE table_group_name = 'tg_replica';

   The result should show one replica per worker node:

   

High availability: continue serving queries when a shard fails

Scenario: Queries fail when the shard they target becomes unavailable.

Configuration:

1. Set replica_count to 2 for the affected table group:

   -- Set replica count to 2 for the tg_replica table group
   CALL hg_set_table_group_property ('tg_replica', 'replica_count', '2');

   The default HA settings apply automatically:

   • hg_experimental_query_replica_mode = leader_follower

   • hg_experimental_query_replica_fixed_plan_ha_mode = any

   • hg_experimental_query_replica_fixed_plan_first_query_timeout_ms = 60

   With these defaults, the system behaves as follows when a shard becomes unavailable:

   Query type	Behavior
   Online analytical processing (OLAP) queries	The master node periodically checks replica availability. Unavailable replicas are removed from routing within 15 seconds (5 s detection + 10 s FE node removal). When the replica recovers, the system adds it back automatically.
   Fixed-plan point queries	A retry mechanism routes failed queries to a replica on another worker node. Queries complete successfully, though response time increases temporarily.
   Write-then-read workloads	Set hg_experimental_query_replica_fixed_plan_ha_mode = leader_first to avoid follower replication lag. Queries go to the leader first; if the leader times out, they fall back to a follower.

   In high-availability mode, QPS is not improved for hot-data queries. To also improve throughput, combine this configuration with a higher replica_count.

2. Verify that replicas are loaded on each worker node:

   SELECT * FROM hologres.hg_worker_info WHERE table_group_name = 'tg_replica';