The numbers of CPU cores and amount of memory resources for Hologres instances vary based on instance specifications. In Hologres, storage resources are independent of computing resources and do not depend on instance specifications. This topic describes the instance specifications that are available for use in Hologres. You can change the specifications of a Hologres instance to meet your business requirements. For example, you can upgrade or downgrade the configurations of the Hologres instance and separately manage its computing and storage resources.
Terms
The resources that are used to run a Hologres instance include process resources consumed for metadata management, computing resources consumed for query services, and resources consumed for importing and caching services for data write optimization. Hologres offers features based on cloud-native container technologies and provides high-performance parallel computing capabilities by using multiple parallel container-based compute nodes.
Hologres determines the maximum number of connections and the shard count for a Hologres instance based on the instance specifications. The default settings are tuned and optimized to meet the requirements of most scenarios. The maximum number of connections cannot be changed, but the shard count can be changed by creating table groups. When a Hologres instance is scaled in or out, the maximum number of connections is automatically adjusted. For databases created before the instance is scaled in or out, the shard count remains unchanged. You must manually change the shard count for the database. For databases created after the instance is scaled in or out, the shard count is determined based on the new instance specifications.
After a Hologres instance is scaled out, more CPU cores are available to improve the performance of concurrent queries. In most cases, you do not need to change the shard count. If you want to write more data to Hologres, you can increase the shard count to improve the write throughput. However, this method does not significantly improve the performance of Online Analytical Processing (OLAP) queries and may even decrease the system throughput. We recommend that you adjust the shard count after you understand the principle. For row-oriented tables, a larger shard count makes Hologres read data from the tables faster.
Recommended instance specifications
In Hologres, data is read from or written to shards. In the same table group, specific data of each table is distributed to the same shard. If you can perform a local join to join multiple tables within the shard, the join efficiency is high. If the data to be joined is not distributed to the same shard, you must use the redistribution operator to shuffle the data. This results in more network transmission and scheduling overheads. Therefore, when you design shards, you must analyze the business scenario and determine whether shard-based high data concurrency or inter-shard data shuffling is required. In data write and update scenarios, data needs to be concurrently written to or updated in multiple shards, and more shards indicate a higher throughput. In point query scenarios, if each query can accurately hit a shard, shard pruning is implemented. In this case, more shards indicate a stronger concurrency capability. For OLAP queries, multiple shards need to participate in the computing, and data exchange is required. An excessive number of shards result in more scheduling overheads between nodes and reduce the concurrency of queries.
When you use a Hologres instance, you may want to estimate the volume of data to store and determine which instance specifications are most appropriate and how many shards need to be configured. However, the instance specifications and the shard count are related to a variety of factors, including the volume of data stored, access frequency, volume of data accessed, compute load type (such as point queries or analysis), write throughput, and the number of tables in a table group. Therefore, it may be difficult for you to determine the appropriate instance specifications and the shard count. The following table describes the recommended instance specifications and shard counts for different data volume ranges. You can select the appropriate settings based on your estimated data volume.
The recommended instance specifications and shard counts for different data volume ranges described in the following table are for reference only. Tables that contain a small volume of data can be added to a table group that has a large shard count, and tables that contain a large volume of data can also be added to a table group that has only one shard. You must select an appropriate shard count based on your business scenario to implement high concurrency, high computing efficiency, and high data concentration, and prevent unnecessary shuffle overheads.
Data volume | Recommended instance specification | Recommended shard count | Description |
Less than 40 million rows | More than 32 cores | 10 to 20 | This configuration is suitable for development environments rather than stress testing. |
40 million to 400 million rows | More than 64 cores | 20 to 40 | This configuration is suitable for scenarios in which no hybrid loads exist. |
400 million to 4 billion rows | More than 128 cores | 40 to 80 | This configuration offers balanced write and query capabilities. We recommend that you use this configuration as starting settings for production systems. |
4 billion to 40 billion rows | More than 256 cores | 80 to 240 | We recommend that you configure multiple table groups and add tables to different table groups based on business cohesion or data amount. Specify different numbers of shards for different table groups and specify a table group for a table when you create the table. |
40 billion to 400 billion rows | More than 512 cores | 160 to 400 | We recommend that you configure multiple table groups and add tables to different table groups based on business cohesion or data amount. Specify different numbers of shards for different table groups and specify a table group for a table when you create the table. Specify a large number of shards for super-large tables. We recommend that you do not specify an excessive number of shards for regular tables. |
Default resource configurations
Hologres determines the maximum number of connections and the shard count for a Hologres instance based on the instance specifications. The following table describes the default resource configurations for each type of instance specifications.
The number of cores for each type of instance specifications includes the numbers of cores for compute nodes and frontend nodes. For instances that have 512 or fewer cores, the number of compute nodes is the same as the number of frontend nodes by default. For instances that have more than 512 cores, the number of frontend nodes is slightly less than the number of compute nodes.
If you want to increase the number of cores for your instance by less than five times, we recommend that you keep the shard count unchanged for the instance. This is because the default shard count is determined based on the performance balance of write and query operations and is applicable to most scenarios.
Each node can have up to 64 GB of memory. The memory is used for computing, caching, and metadata storage. For Hologres instances earlier than V1.1.24, each node can have up to 20 GB of memory for computing during runtime. For Hologres instances of V1.1.24 or later, the memory for computing is elastically allocated during runtime. Remaining memory that is not used is allocated for computing to improve the overall memory usage.
The maximum number of connections for an instance is calculated by using the following formula: Maximum number of connections for a frontend node × Number of frontend nodes.
Instance specification | Default number of compute nodes | Default shard count (for Hologres V0.10.31 and later) | Maximum number of connections (for Hologres V0.10.25 and later) | Number of connections reserved for the superuser (for Hologres V1.1 and later) |
32 cores | 2 | 20 | 256 (128 × 2) | 10 (5 × 2) |
64 cores | 4 | 40 | 512 (128 × 4) | 20 (5 × 4) |
96 cores | 6 | 60 | 768 (128 × 6) | 30 (5 × 6) |
128 cores | 8 | 80 | 1,024 (128 × 8) | 40 (5 × 8) |
160 cores | 10 | 80 | 1,280 (128 × 10) | 50 (5 × 10) |
192 cores | 12 | 80 | 1,536 (128 × 12) | 60 (5 × 12) |
256 cores | 16 | 120 | 2,048 (128 × 16) | 80 (5 × 16) |
384 cores | 24 | 160 | 3,072 (128 × 24) | 120 (5 × 24) |
512 cores | 32 | 160 | 4,096 (128 × 32) | 160 (5 × 32) |
New instance specifications
From April 25, 2022, Hologres supports instances that have 640 to 1024 cores. If you need to create a Hologres instance that has more than 1,024 cores, manually upgrade your Hologres instance in the Hologres console or join a Hologres DingTalk group to apply for an instance upgrade. For more information about how to manually upgrade a Hologres instance, see Instance upgrades. For more information about how to join a Hologres DingTalk group, see Obtain online support for Hologres. Before you scale out your instance, make sure that the version of your Hologres instance is V1.1.58 or later. The following table describes the default resource configurations for each type of new instance specifications.
Instance specification | Default number of compute nodes | Default shard count (for Hologres V1.1.58 and later) | Maximum number of connections (for Hologres V1.1.58 and later) | Number of connections reserved for the superuser (for Hologres V1.1.58 and later) |
640 cores | 40 | 160 | 5,120 (128 × 40) | 200 (5 × 40) |
768 cores | 48 | 160 | 6,144 (128 × 48) | 240 (5 × 48) |
896 cores | 56 | 160 | 7,168 (128 × 56) | 280 (5 × 56) |
1,024 cores | 64 | 200 | 8,192 (128 × 64) | 320 (5 × 64) |
1,280 cores | 80 | 200 | 10,240 (128 × 80) | 400 (5 × 80) |
1,536 cores | 96 | 200 | 12,288 (128 × 96) | 480 (5 × 96) |
1,792 cores | 112 | 200 | 14,336 (128 × 112) | 560 (5 × 112) |
2,048 cores | 128 | 200 | 16,384 (128 × 128) | 640 (5 × 128) |
2,304 cores | 144 | 240 | 18,432 (128 × 144) | 720 (5 × 144) |
2,560 cores | 160 | 240 | 20,480 (128 × 160) | 800 (5 × 160) |
3,072 cores | 192 | 240 | 24,576 (128 × 192) | 960 (5 × 192) |
3,584 cores | 224 | 240 | 28,672 (128 × 224) | 1,120 (5 × 224) |
4,096 cores | 256 | 320 | 32,768 (128 × 256) | 1,280 (5 × 256) |
4,608 cores | 288 | 320 | 36,864 (128 × 288) | 1,440 (5 × 288) |
5,120 cores | 320 | 320 | 40,960 (128 × 320) | 1,600 (5 × 320) |
5,632 cores | 352 | 320 | 45,056 (128 × 352) | 1,760 (5 × 352) |
6,144 cores | 384 | 320 | 49,152 (128 × 384) | 1,920 (5 × 384) |
6,656 cores | 416 | 320 | 53,248 (128 × 416) | 2,080 (5 × 416) |
7,168 cores | 448 | 320 | 57,344 (128 × 448) | 2,240 (5 × 448) |
7,680 cores | 480 | 320 | 61,440 (128 × 480) | 2,400 (5 × 480) |
8,192 cores | 512 | 400 | 65,536 (128 × 512) | 2,560 (5 × 512) |
Query the maximum number of connections and manage connections
This section describes how to query the maximum number of connections for a Hologres instance and manage the connections to the Hologres instance.
Query the maximum number of connections for a Hologres instance.
After you create a Hologres instance and connect to the instance by using a development tool, you can execute the following statement to query the maximum number of connections for a single frontend node.
NoteThe maximum number of connections for a Hologres instance is calculated by using the following formula: Maximum number of connections for a single frontend node × Number of frontend nodes.
-- Query the maximum number of connections for a single frontend node. The connections are distributed among multiple frontend nodes. show max_connections;Manage connections.
Hologres reserves connections for superusers of an instance. When the number of connections to the instance reaches the upper limit specified for the instance specifications, the superusers can connect to Hologres and execute SQL statements to query idle connections and release them. The superusers can also scale out the instance based on the business requirements. For more information about how to query and release idle connections, see the "Connections" section of the Hologres metrics topic.
Query and change the shard count for a Hologres instance
You can increase the number of cores for a Hologres instance to achieve higher concurrent query capabilities. In most cases, you do not need to adjust the shard count after you increase the number of cores. If you want to write more data to Hologres, you can increase the shard count to improve the concurrent write throughput.
For row-oriented tables, a larger shard count makes Hologres read data from the tables faster. For more information about how to query and change the shard count for a Hologres instance, see User guide of table groups and shard counts.