Hologres:Hologres metrics

On the Monitoring Information page for a Hologres instance in the Hologres console, you can view metrics about instance resources and query execution. You can view the resource usage of the instance and the execution status of SQL statements. This allows you to manage and monitor the instance at any time, identify system bottlenecks, and handle instance exceptions at the earliest opportunity. In September 2023, the underlying technology of the monitoring service in the Hologres console was upgraded. After the upgrade, metrics are reported based on the following rules:

• Metrics of the current day are aggregated and reported at an interval of 1 minute.

• Metrics of the previous 5 days are aggregated and reported at an interval of 5 minutes.

• Metrics of the previous 10 days are aggregated and reported at an interval of 10 minutes.

• Metrics of the previous 15 days are aggregated and reported at an interval of 15 minutes.

• Metrics of the previous 30 days are aggregated and reported at an interval of 30 minutes.

The CPU Utilization (%) metrics include CPU Utilization (%) - Instance and CPU Utilization (%) - Worker Nodes.

• CPU Utilization (%) - Instance

  The CPU Utilization (%) - Instance metric indicates the overall CPU utilization of an instance, and reflects the computing load of the instance.

  • The daemon processes that are running in the background or the compaction task that is executed asynchronously may occupy the CPU of the instance. Therefore, a small amount of CPU utilization is normal when no query tasks are running in the instance.

  • Hologres supports multi-core parallel computing. In most cases, the CPU utilization during a single query can increase to 100%. This indicates that the computing resources are being fully utilized.

  • If the CPU utilization remains close to 100%, the instance is heavily loaded. For example, the CPU utilization remains 100% for 3 consecutive hours or remains 90% and higher for 12 consecutive hours. In this case, CPU resources have become a bottleneck of the instance. You must analyze specific business scenarios and queries to identify the cause of the high CPU utilization. You can identify the cause from the following aspects:

    • Check whether a large amount of offline data is imported by executing the INSERT statement, and whether the amount of data is still increasing.

    • Check whether data is queried or written with high queries per second (QPS) to exhaust the CPU resources.

    • Check whether hybrid loads exist in the preceding or other scenarios.

  • Scale the instance to accommodate more complex queries or larger data volumes.

• CPU Utilization (%) - Worker Nodes

  The CPU Utilization (%) - Worker Nodes metric indicates the CPU utilization of each worker node in an instance, and reflects the computing load of each worker node. The number of worker nodes allocated to a Hologres instance varies based on the instance type. For more information about instance types, see Instance types.

  Note

  This metric is added in July 2022.

  • This metric is displayed only for Hologres instances of V1.1 and later.

  • If the CPU utilization of all worker nodes remains close to 100%, the instance is heavily loaded. You must optimize the resource usage or scale the instance based on the specific business scenario.

  • If the CPU utilization of some worker nodes is high and the CPU utilization of other worker nodes is low, the resources are unevenly allocated to the worker nodes. For more information about common causes for uneven resource allocation and troubleshooting methods, see Metric FAQ.

The Storage Usage (bytes) metric indicates the size of the logical disk that is used to store the data of all databases in an instance. 1 GiB = 1,024 MiB = 1,024 × 1,024 KiB. The storage usage of a Hologres instance can be continuously increased without limits.

• After the storage quota of a subscription instance is used up, the excess storage is charged in pay-as-you-go mode. Excess storage usage does not affect the stability or use of the Hologres instance. We recommend that you update storage configurations or clean up unwanted data at the earliest opportunity after your storage quota is used up. This prevents unnecessary and more expensive storage costs.

• You can execute the following SQL statements to query the storage usage of each database and internal tables.

  Note

  The unit of the execution results of the pg_relation_size and pg_database_size functions is byte. The unit of the execution results of the pg_size_pretty function is KB, MB, or GB. The unit is automatically converted based on the data volume. In the unit conversion standard of PostgreSQL, 1 GB = 1,024 MB = 1,024 × 1,024 KB.

  select pg_database_size('dbname'); -- Query the storage size of the database.
  select pg_relation_size('tablename'); -- Query the storage size of the table.
  
  -- Sort all tables by the storage size.
  select relname as "Table",
         pg_relation_size(relname)/1024/1024/1024 as "Size(GB)"
  from pg_tables
  where schemaname = 'public'
  order by pg_relation_size(relname)/1024/1024/1024;

The Connections metrics include Connections - Instance and Connections - FEs. The default maximum number of connections supported by an Hologres instance varies based on the instance type. For more information about instance types, see Instance types.

• Connections - Instance

  The Connections - Instance metrics include Used Connections and Highest FE Connection Usage (%).

  • Used Connections: the total number of connections in an instance, including Java Database Connectivity (JDBC) and PostgreSQL connections in different states, such as active connections and idle connections.

  • Highest FE Connection Usage (%): the highest connection usage among FE nodes. Hologres collects the connection usage of each FE node and displays the highest connection usage by using the Max(frontend_connection_used_rate) function. This allows you to check whether the number of connections exceeds the maximum number supported by an FE node to prevent connection failures.

    Note

    This metric is added in July 2022.

    • The round robin policy is adopted among multiple FE nodes to establish physical connections to these FE nodes in sequence.

    • If the number of connections is large or even exceeds the maximum number of connections supported by your instance, or one of the following errors occurs, the number of connections in the Hologres instance has reached the upper limit. In this case, you must check whether a connection leak occurs. If a connection leak occurs, release connections by following instructions in Manage connections.

      FATAL: sorry, too many clients already connection limit exceeded for superusers 
      
      FATAL: remaining connection slots are reserved for non-replication superuser connections

• Connections - FEs

  The Connections - FEs metric indicates the number of connections used by each FE node in an instance, including JDBC and PostgreSQL connections in different states, such as active connections and idle connections.

  Note

  This metric is added in July 2022.

  • This metric is displayed only for Hologres instances of V1.1 and later.

  • By default, an FE node supports a maximum of 128 connections. For more information, see Instance types.

  • If the number of connections used by each FE node in an instance remains close to the maximum number, a large number of connections exist in the instance. You can check whether the connections are idle or active based on your business scenario. For more information, see Manage connections. Then, release idle connections or scale the instance at the earliest opportunity based on your business scenario.

  • If the number of connections used by some worker nodes is high whereas the number of connections used by other worker nodes is low, the connections of the instance are unevenly distributed. You can find idle connections and release the idle connections at the earliest opportunity to mitigate the uneven connection load.

The SDK RPS indicates the number of entries that are imported or updated per second by using fixed plans. Data can be imported by using SDKs in the following ways:

• Accelerate the execution of SQL statements by using fixed plans.

• Use Flink to write data to Hologres.

• Use Holo Client to read and write data.

• Use Spark to write data to Hologres.

• Synchronize data from DataHub to Hologres in real time.

• Use the INSERT INTO VALUES () statement to write data by using the PostgreSQL client or a JDBC client.

The Ongoing Query Duration metric indicates the duration of an ongoing query in an instance. By default, this metric displays the duration of the ongoing query that is the longest at the current point in time.

• This metric is displayed only for Hologres instances of V1.1 and later.

• Hologres is a distributed system. The numbers of worker nodes that are configured on an Hologres instance varies based on the instance type. Queries are randomly distributed to worker nodes. Hologres collects the durations of ongoing queries on each worker node and displays the duration of the ongoing query that is the longest at the current point in time. For example, three queries are being executed on different worker nodes at the current point in time, and the durations of these queries are 10 minutes, 5 minutes, and 30 seconds. In this case, the Ongoing Query Duration metric displays 10 minutes.

• You can check whether the duration of the query is normal based on logs of active queries or slow queries. For more information, see Manage queries and Query and analyze slow query logs. Troubleshoot slow queries based on the preceding topics, and resolve deadlocks or stuck issues at the earliest opportunity.

The Failed Queries per Second metric indicates the average number of failed SQL statements per second in an instance, including the SELECT, INSERT, UPDATE, DELETE, and OTHER statements.

• This metric is updated every 20 seconds. If only a single SQL statement fails within 20 seconds, the failure rate of SQL statements is 0.05 (1/20 = 0.05).

• You can view the failed queries in slow query logs and troubleshoot the failures to improve the availability of your Hologres instance. For more information, see Query and analyze slow query logs.

Hologres provides the multi-instance high-availability deployment mode. You can configure four read-only secondary instances for a primary instance. For more information, see Configure read/write splitting for primary and secondary instances (shared storage). The Primary/Secondary Latency (ms) metric indicates the latency of data synchronization from a primary instance to secondary instances. The latency is displayed in milliseconds.

• Only Hologres instances of V1.1 and later support primary/secondary deployment. This metric is displayed only for secondary instances. By default, this metric is not displayed for primary instances.

• This metric is initially displayed as 0 ms after a secondary instance is associated with a primary instance. When data is written to the primary instance, this metric displays the fluctuating latency of data synchronization to the secondary instance.

• In most cases, the synchronization latency between the primary and secondary instances is in milliseconds. The occasional latency jitters are usually caused by metadata modification operations, such as DDL operations, that are performed on the primary instance. These latency jitters can be ignored. If the latency is greater than one second in a long period of time, the possible cause is that the resource usage is high in the instances. You can check the resource usage such as CPU utilization and memory usage and scale the instances if necessary to reduce the latency. If an instance is restarting or updating, the synchronization latency may increase to minutes and will automatically recover after the restart or update is complete.