In Hologres, the number of CPU cores and the amount of memory resources for instances vary based on instance specifications. Storage resources are independent of computing resources and do not depend on instance specifications. This topic describes the instance specifications that are available for use in Hologres. You can change the specifications of a Hologres instance to meet your business requirements. For example, you can upgrade or downgrade the configurations of the Hologres instance and separately manage its computing and storage resources.
Basic concepts
The runtime resources of Hologres include process resources for metadata management, computing resources for query services, import pipeline resources for optimizing data writes, and caching services. Hologres offers features based on cloud-native container technologies and provides high-performance parallel computing capabilities by using multiple parallel container-based compute nodes.
Hologres determines the maximum number of connections and the shard count for a Hologres instance based on the instance specifications. The default settings are tuned and optimized to meet the requirements of most scenarios. The maximum number of connections cannot be changed, but the shard count can be changed by creating table groups. When a Hologres instance is scaled in or out, the maximum number of connections is automatically adjusted. For databases created before the instance is scaled in or out, the shard count remains unchanged. You must manually change the shard count for the database. For databases created after the instance is scaled in or out, the shard count is determined based on the new instance specifications.
After a Hologres instance is scaled out, more CPU cores are available to improve the performance of concurrent queries. In most cases, you do not need to change the shard count. If you want to write more data to Hologres, you can increase the shard count to improve the write throughput. However, this method does not significantly improve the performance of Online Analytical Processing (OLAP) queries and may even decrease the system throughput. We recommend that you adjust the shard count after you understand the principle. For row-oriented tables, a larger shard count makes Hologres read data from the tables faster.
Recommended instance specifications
In Hologres, data is read from or written to shards. In the same table group, specific data of each table is distributed to the same shard. If you can perform a local join to join multiple tables within the shard, the join efficiency is high. If the data to be joined is not distributed to the same shard, you must use the redistribution operator to shuffle the data. This results in more network transmission and scheduling overheads. Therefore, when you design shards, you must analyze the business scenario and determine whether shard-based high data concurrency or inter-shard data shuffling is required. In data write and update scenarios, data can be concurrently written to or updated in multiple shards. Therefore, the more the number of shards, the higher the throughput. In point query scenarios, if each query can accurately hit a shard, shard pruning is implemented. In this case, more shards indicate a stronger concurrency capability. For OLAP queries, multiple shards need to participate in the computing, and data exchange is required. An excessive number of shards will result in more scheduling overheads between nodes and reduce the concurrency of queries.
When you use a Hologres instance, you may want to estimate the volume of data to store and determine which instance specifications are most appropriate and how many shards need to be configured. However, the instance specifications and the shard count are related to a variety of factors, including the volume of data stored, access frequency, volume of data accessed, compute load type (such as point queries or analysis), write throughput, and the number of tables in a table group. Therefore, it may be difficult for you to determine the appropriate instance specifications and the shard count. The following table describes the recommended instance specifications and shard counts for different data volume ranges. You can select the appropriate settings based on your estimated data volume.
The recommended instance specifications and shard counts for different data volume ranges described in the following table are for reference only. Tables that contain a small volume of data can be added to a table group that has a large shard count, and tables that contain a large volume of data can also be added to a table group that has only one shard. You must select an appropriate shard count based on your business scenario to implement high concurrency, high computing efficiency, and high data concentration, and prevent unnecessary shuffle overheads.
Data volume | Recommended instance specification | Recommended shard count | Description |
Less than 40 million rows | More than 32 cores | 10 to 20 | This configuration is suitable for development environments rather than stress testing. |
40 million to 400 million rows | More than 64 cores | 20 to 40 | This configuration is suitable for scenarios in which no hybrid loads exist. |
400 million to 4 billion rows | More than 128 cores | 40 to 80 | This configuration offers balanced write and query capabilities. We recommend that you use this configuration as starting settings for production systems. |
4 billion to 40 billion rows | More than 256 cores | 80 to 240 | We recommend that you configure multiple table groups and add tables to different table groups based on business cohesion or data amount. Specify different numbers of shards for different table groups and specify a table group for a table when you create the table. |
40 billion to 400 billion rows | More than 512 cores | 160 to 400 | We recommend that you configure multiple table groups and add tables to different table groups based on business cohesion or data amount. Specify different numbers of shards for different table groups and specify a table group for a table when you create the table. Specify a large number of shards for super-large tables. We recommend that you do not specify an excessive number of shards for regular tables. |
Default resource configurations
Hologres determines the maximum number of connections and the shard count for a Hologres instance based on the instance specifications. The following table describes the default resource configurations for each type of instance specifications.
Starting from April 25, 2022, general-purpose instances support the specifications of 512 to 1,024 compute units (CUs). If you need higher specifications, submit a ticket. Before scaling up to higher specifications, upgrade your instance to Hologres V1.1.58 or later.
You can flexibly purchase a virtual warehouse instance with specifications of 32 to 8,192 CUs without the need to submit a ticket.
Each instance contains compute nodes and frontend nodes. Each compute node corresponds to 16 CUs. In instances with 512 or fewer cores, the number of compute nodes is the same as the number of frontend nodes by default. In instances with 1,600 or more CUs, the number of frontend nodes is fixed to 100.
If you want to increase the specifications of your instance by less than five times, we recommend that you keep the shard count unchanged for the instance. This is because the default shard count is determined based on the performance balance of write and query operations and is applicable to most scenarios.
The maximum number of connections for an instance is calculated by using the following formula: Maximum number of connections for a frontend node × Number of frontend nodes.
Instance specification | Number of compute nodes | Default shard count | Maximum number of connections (for Hologres V2.1 and earlier) | Maximum number of connections (for Hologres V2.2 and later) | Number of connections reserved for the superuser (for Hologres V1.1 and later) |
32 CUs | 2 | 20 | 256 (128 × 2) | 512 (256 × 2) | 10 (5 × 2) |
48-80 CUs | 3-5 | 40 | 128 × Number of compute nodes | 256 × Number of compute nodes | 5 × Number of compute nodes |
96-112 CUs | 6-7 | 60 | 128 × Number of compute nodes | 256 × Number of compute nodes | 5 × Number of compute nodes |
128-192 CUs | 8-12 | 80 | 128 × Number of compute nodes | 256 × Number of compute nodes | 5 × Number of compute nodes |
208-352 CUs | 13-22 | 120 | 128 × Number of compute nodes | 256 × Number of compute nodes | 5 × Number of compute nodes |
368-992 CUs | 23-62 | 160 | 128 × Number of compute nodes | 256 × Number of compute nodes | 5 × Number of compute nodes |
1008-1584 CUs | 63-99 | 200 | 128 × Number of compute nodes | 256 × Number of compute nodes | 5 × Number of compute nodes |
1600-2272 CUs | 100-142 | 200 | 12800 (128 × 100) | 25600 (256 × 100) | 500 (5 × 100) |
2288-4000 CUs | 143-250 | 240 | 12800 (128 × 100) | 25600 (256 × 100) | 500 (5 × 100) |
4016-8000 CUs | 251-500 | 320 | 12800 (128 × 100) | 25600 (256 × 100) | 500 (5 × 100) |
8016-8192 CUs | 501-512 | 400 | 12800 (128 × 100) | 25600 (256 × 100) | 500 (5 × 100) |
Query the maximum number of connections and manage connections
This section describes how to query the maximum number of connections for a Hologres instance and manage the connections to the Hologres instance.
Query the maximum number of connections for a Hologres instance.
After you create a Hologres instance and connect to the instance by using a development tool, you can execute the following statement to query the maximum number of connections for a single frontend node.
NoteThe maximum number of connections for a Hologres instance is calculated by using the following formula: Maximum number of connections for a single frontend node × Number of frontend nodes.
-- Query the maximum number of connections for a single frontend node. The connections are evenly distributed among multiple frontend nodes. show max_connections;Manage connections.
Hologres reserves connections for superusers of an instance. When the number of connections to the instance reaches the upper limit specified for the instance specifications, the superusers can connect to Hologres and execute SQL statements to query idle connections and release them. The superusers can also scale out the instance based on the business requirements. For more information about how to query and release idle connections, see the "Connections" section of the Hologres metrics topic.
Query and change the shard count for a Hologres instance
You can increase the number of cores for a Hologres instance to achieve higher concurrent query capabilities. In most cases, you do not need to adjust the shard count after you increase the number of cores. If you want to write more data to Hologres, you can increase the shard count to improve the concurrent write throughput.
For row-oriented tables, a larger shard count makes Hologres read data from the tables faster. For more information about how to query and change the shard count for a Hologres instance, see User guide of table groups and shard counts.