Hologres:Hologres metrics

Instance CPU Usage(%)

This metric indicates the overall CPU utilization of an instance, and reflects the computing load of the instance.

• Daemon processes that run in the background or a compaction task that is asynchronously executed may occupy the CPU of the instance. Therefore, a small amount of CPU utilization is normal when no query tasks are running in the instance.

• Hologres supports multi-core parallel computing. In most cases, the CPU utilization during a single query can quickly increase to 100%. This indicates that the computing resources are being fully utilized.

• If the CPU utilization remains close to 100%, the instance is heavily loaded. For example, the CPU utilization remains 100% for 3 consecutive hours or remains 90% and higher for 12 consecutive hours. In this case, CPU resources have become a bottleneck of the instance. You must analyze specific business scenarios and queries to identify the cause of the high CPU utilization. You can identify the cause by considering the following aspects:

  • Check whether a large amount of offline data is imported by executing the INSERT statement, and whether the amount of data is still increasing.

  • Check whether data is queried or written at a high queries per second (QPS), which exhausts the CPU resources.

  • Check whether hybrid loads exist in the preceding or other scenarios.

• Scale out the instance to accommodate more complex queries or larger data volumes.

Worker CPU Usage(%)

This metric indicates the CPU utilization of each worker node in an instance, and reflects the computing load of each worker node. The number of worker nodes allocated to a Hologres instance varies based on instance specifications. For more information about instance specifications, see Instance specifications.

• This metric is displayed only in Hologres V1.1 and later.

• If the CPU utilization of all worker nodes remains close to 100%, the instance is heavily loaded. In this case, you must optimize the resource usage or scale out the instance based on the specific business scenario.

• If the CPU utilization of some worker nodes is high and the CPU utilization of other worker nodes is low, the resources are unevenly allocated to the worker nodes. For more information about common causes of uneven resource allocation and their troubleshooting methods, see Metric FAQ.

Instance Memory Usage(%)

This metric indicates the overall memory usage of an instance.

• In Hologres, memory resources are reserved by backend processes. Even if no queries are in progress, the metadata, indexes, and data cache of data tables are loaded to the memory to facilitate archiving and computing. Therefore, the memory usage is not zero. When no queries are in progress, memory usage of 30% to 40% is normal.

• However, if the memory usage continues to increase and remains close to 80%, memory resources may become a bottleneck of the instance. This can have a negative impact on the stability or performance of the instance.

• You can identify the components or processes that occupy a large amount of memory resources based on the Memory Usage Detail(%) metric and QPS of your business. This information helps you know where you can make optimizations. For more information, see FAQ about OOM.

Worker Memory Usage(%)

This metric indicates the memory usage of each worker node in an instance, and reflects the computing load of each worker node. The number of worker nodes allocated to a Hologres instance varies based on instance specifications. For more information about instance specifications, see Instance specifications.

• This metric is displayed only in Hologres V1.1 and later.

• If the memory usage of all worker nodes remains close to 80%, the instance is heavily loaded. In this case, you must optimize the resource usage or scale out the instance based on business requirements.

• If the memory usage of some worker nodes is high and the memory usage of other worker nodes is low, the resources are unevenly allocated to worker nodes. For more information about common causes of uneven resource allocation and their troubleshooting methods, see Metric FAQ.

Memory Usage Detail(%)

This metric provides memory usage data for the following components and processes in an instance: system, metadata, cache, query, and background. This metric is displayed only in Hologres V2.0.15 and later. This metric provides the memory distribution of an instance, based on which you can perform performance optimization.

• System: indicates the memory resources that are occupied by system components during runtime. The system components include the HoloHub, gateways, and FE nodes. FE nodes include FE master nodes and FE query nodes. The memory resources that are occupied by system components fluctuate when queries are performed.

• Cache: indicates the memory resources that are occupied by the following cached data:

  • Cached SQL-related data: includes cached results and cached blocks. The memory resources that are occupied by cached SQL-related data fluctuate based on the query running status. If an SQL query hits the cache, the query performance is improved. For example, you can execute the EXPLAIN ANALYZE statement to obtain the value of the Physical read bytes field. If the value of this field is small, the SQL query hits a large amount of cache resources. The amount of cached SQL-related data is limited.

  • Cached metadata: includes cached metadata of table schemas and cached metadata of files. The storage engine of Hologres loads metadata to the cache before SQL queries. This avoids direct access to disks and improves query performance.

  • The memory resources that are occupied by cached data is fixed. In most cases, about 30% of the total memory of an instance is occupied by cached data. If no queries are performed on an instance, cached metadata occupies the memory.

• Metadata: indicates the memory that is occupied by metadata and files. The Lazy Open mode is used for storing metadata. In this mode, frequently accessed metadata is stored in memory, and metadata that is not accessed is not stored in memory. This helps reduce the memory that is occupied by metadata. In most cases, we recommend that less than 30% of the total memory of an instance be occupied by metadata. If metadata occupies a large amount of memory resources in an instance, a large number of files or partitioned tables in the instance are accessed. In this case, you can perform table governance in the instance based on table statistics. For more information, see Query and analyze table statistics.

• Query: indicates the memory resources that are consumed when SQL statements are executed. The amount of memory resources consumed is positively correlated with the complexity and concurrency of queries. Memory resources are consumed when SQL statements are executed by using fixed plans, the HQE, or the SQE.

  • The memory that is available for SQL queries is elastic. A minimum of 20 GB of memory is required by a worker node. The maximum available memory for a worker node varies based on the total available memory of the instance. If a large amount of memory is used by other components and processes, a small amount of memory is available for computing.

  • If queries on an instance occupy a large amount of memory or an out of memory (OOM) issue occurs on the instance, the queries are complex or have high concurrency. We recommend that you optimize the query performance or scale out the instance to prevent high memory usage. For more information, see Optimize performance of queries on Hologres internal tables.

• Background: indicates the memory resources that are occupied by background tasks, including compaction and flushing tasks. In most cases, background tasks occupy less than 5% of the total memory of an instance. If table indexes are modified, or a large amount of data is written or updated, background tasks occupy more memory resources until the tasks are complete.

• Memtable: indicates the memory resources that are occupied by memory tables. In Hologres, a memory table is used to store table data that is written, updated, or deleted in real time. In most cases, memory tables occupy less than 5% of the total memory of an instance.

QE Memory Used(byte)

This metric indicates the amount of memory resources that are used by XQE-based queries. XQE-based queries indicate queries that use Alibaba Cloud-developed engines, such as HQE or SQE.

• This metric is displayed only in Hologres V2.0.44 and later and V2.1.22 and later.

• The amount of memory resources that are used by queries includes the value of this metric.

This metric reflects the complexity of queries that run in an instance. If the value of this metric is large, queries that run in the instance are complex, and a large amount of memory resources are used.

QE Memory Used percentage(%)

This metric reflects the instance load. If the value of this metric is high, an OOM issue may occur due to insufficient memory. In this case, you must perform query governance or scale out your instance.

This metric is displayed only in Hologres V2.0.44 and later and V2.1.22 and later.

Query QPS(countS)

This metric indicates the total number of queries that are performed by using SQL statements per second in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, DELETE, UTILITY, and UNKNOWN types. Query QPS ≥ QE QPS + FixedQE QPS

QE QPS(countS)

This metric indicates the number of QE-based queries that are performed by using SQL statements per second in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, and DELETE types.

This metric is displayed only in Hologres V2.2 and later.

FixedQE QPS(countS)

This metric indicates the number of FixedQE-based queries that are performed by using SQL statements per second in an instance. FixedQE is named SDK in versions earlier than Hologres V2.2. The SQL statements include statements of the SELECT, INSERT, UPDATE, and DELETE types.

This metric is displayed only in Hologres V2.2 and later.

DML RPS(countS)

This metric indicates the total number of data records that are imported or updated in queries that are performed by using DML statements per second in an instance. The DML statements include statements of the INSERT, UPDATE, and DELETE types. DML RPS = QE DML RPS + Fixed QE DML RPS.

QE DML RPS(countS)

This metric indicates the number of data records that are imported or updated in QE-based queries that are performed by using DML statements per second in an instance. The DML statements include statements of the INSERT, UPDATE, and DELETE types.

• This metric is displayed only in Hologres V2.2 and later.

• QE is commonly used to import or update data in the following scenarios:

  • Batch import or update of data in MaxCompute or OSS tables

  • Data writes or updates in batch copy mode

  • Batch import between Hologres tables

Fixed QE DML RPS(countS)

This metric indicates the number of data records that are imported or updated in FixedQE-based queries that are performed by using DML statements per second in an instance. The DML statements include statements of the INSERT, UPDATE, and DELETE types. FixedQE is named SDK in versions earlier than Hologres V2.2.

• This metric is displayed only in Hologres V2.2 and later.

• FixedQE is commonly used in the following scenarios:

  • Real-time data writes by using Flink

  • Data writes by using Holo Client

  • Data writes to Hologres by using Spark

  • Real-time synchronization of MySQL binary logs to Hologres

  • Real-time synchronization of DataHub data to Hologres

  • Batch data writes by using DataX of Data Integration

  • Data writes by executing the INSERT INTO VALUES () statement on the PostgreSQL client or JDBC client

Query Latency(milliseconds)

This metric indicates the average latency of all queries that are performed by using SQL statements in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, DELETE, UTILITY, and UNKNOWN types. Query Latency ≥ MAX(QE Latency, FixedQE Latency)

QE Latency(milliseconds)

This metric indicates the average latency of QE-based queries that are performed by using SQL statements in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, and DELETE types.

• This metric is displayed only in Hologres V2.2 and later.

• You can use the QE Latency(milliseconds) metric together with the Optimization Cost Latency(milliseconds), Start Query Cost Latency(milliseconds), Get Next Cost Latency(milliseconds), and QE QPS(countS) metrics.

FixedQE Latency(milliseconds)

This metric indicates the average latency of FixedQE-based queries that are performed by using SQL statements in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, and DELETE types.

• This metric is displayed only in Hologres V2.2 and later.

• The high value of this metric may be caused by the following reasons:

  • Occasionally high: HQE-based queries may exist and acquire locks. If the value of the FixedQE Lock Latency metric also increases, HQE-based queries acquire locks. In this case, identify the queries that acquire locks by following the instructions in Query Insight.

  • Always high: Table configurations may be inappropriate, or other complex queries exist. Troubleshoot the fault by following the instructions in FAQ about Blink and Flink issues.

Optimization Cost Latency(milliseconds)

This metric indicates the duration of queries in an instance in the optimization phase. In the optimization phase, the query optimizer (QO) parses SQL statements and generates physical plans to help the execution engine to process data.

• This metric is displayed only in Hologres V2.0.44 and later and V2.1.22 and later.

• If the value of this metric is high, queries in the instance are complex, and query parsing is time-consuming. If queries in the instance differ only in their input parameters, we recommend that you use prepared statements to shorten the optimization phase. For more information, see JDBC.

Start Query Cost Latency(milliseconds)

This metric indicates the duration of queries in an instance in the start query phase. In this phase, pre-query initialization is performed, such as lock acquisition and schema version alignment.

• This metric is displayed only in Hologres V2.0.44 and later and V2.1.22 and later.

• If the value of this metric is high, the time that SQL statements spend on waiting for locks to be released or waiting for resources due to the high CPU load is long. You can perform troubleshooting based on the execution plan.

Get Next Cost Latency(milliseconds)

This metric indicates the duration from the end of the start query phase to the time when all results are returned. In this phase, the system processes data and returns results.

• This metric is displayed only in Hologres V2.0.44 and later and V2.1.22 and later.

• If the value of this metric is high, queries are complex. You can check values of the QE Memory Used percentage(%) and QE QPS(countS) metrics. If the values of the metrics are normal, the high value of this metric may be caused by the long time the client spends on waiting for data to be returned.

Query Latency P99(milliseconds)

This metric indicates the 99th percentile latency of all queries in an instance. The queries include system queries and queries that are performed by using statements of the SELECT, INSERT, UPDATE, and UTILITY types.

Longest Active Query Time(milliseconds)

This metric indicates the execution time of the ongoing query with the longest duration in an instance at the current point in time. The query statements include statements of the SELECT, INSERT, UPDATE, DELETE, UTILITY, and UNKNOWN types.

• This metric is displayed only in Hologres V1.1 and later.

• The number of worker nodes that are configured on an instance varies based on instance types. Queries are randomly distributed to worker nodes. Hologres collects the durations of ongoing queries on each worker node and displays the duration of the longest ongoing query at the current point in time. For example, if three queries are being executed on different worker nodes at the current point in time, and the durations of these queries are 10 minutes, 5 minutes, and 30 seconds, this metric displays 10 minutes.

• You can check whether the duration of the query is normal based on logs of active queries or slow queries. For more information, see Manage queries and Query and analyze slow query logs. Troubleshoot slow queries, and resolve deadlocks or stuck issues at the earliest opportunity.

Failed Query Qps(countS)

This metric indicates the average number of failed queries that are performed by using SQL statements per second in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, DELETE, UTILITY, and UNKNOWN types. Failed Query QPS ≥ Failed QE QPS + Failed FixedQE QPS.

You can view the failed queries in slow query logs and troubleshoot the failures to improve the availability of your Hologres instance. For more information, see Query and analyze slow query logs.

Failed QE QPS(countS)

This metric indicates the average number of failed QE-based queries that are performed by using SQL statements per second in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, and DELETE types.

This metric is displayed only in Hologres V2.2 and later.

Failed FixedQE QPS(countS)

This metric indicates the average number of failed FixedQE-based queries that are performed by using SQL statements per second in an instance. The SQL statements include statements of the SELECT, INSERT, UPDATE, and DELETE types.

This metric is displayed only in Hologres V2.2 and later.

FE Wait Lock Latency(milliseconds)

Hologres is a distributed system. A Hologres instance has multiple FE nodes that parse, distribute, and route SQL statements. An FE node is locked if multiple connection requests are routed to the FE node for DDL operations on the same table. These DDL operations include the CREATE and DROP operations. This metric indicates the longest time that DDL statements spend on waiting for FE locks to be released.

• This metric is displayed only in Hologres V2.2 and later.

• In most cases, a DDL statement acquires a lock when the statement is executed, and other statements need to wait for the lock to be released. If the value of this metric exceeds 5 minutes, and the value of the FE Replay Running Time(milliseconds) metric is large, the execution of specific DDL statements may be stuck. You can identify and terminate time-consuming queries by following the instructions in Manage queries.

FixedQE Lock Latency(milliseconds)

SQL statements of the INSERT, DELETE, and UPDATE types that use HQE acquire table locks, and SQL statements of the INSERT, DELETE, and UPDATE types that use FixedQE acquire row locks. As a result, FixedQE-based SQL statements need to wait for locks acquired by HQE-based SQL statements to be released, and the value of this metric increases.

• This metric is displayed only in Hologres V2.2 and later.

• If the value of this metric is large, identify time-consuming FixedQE-based queries based on slow query logs and check the HQE-based SQL statements that acquire locks by using the Query Insight feature.

BE Lock Latency(milliseconds)

This metric indicates the total duration that SQL statements of the INSERT, DELETE, and UPDATE types spend on acquiring locks. The SQL statements include FixedQE-based statements and HQE-based statements.

• This metric is displayed only in Hologres V2.2 and later.

• If the value of this metric is large, identify time-consuming SQL statements of the INSERT, DELETE, and UPDATE types based on slow query logs and check HQE-based statements that acquire locks by using the Query Insight feature.

Connections(Count)

The default maximum number of connections supported by a Hologres instance varies based on instance specifications. For more information about instance specifications, see Instance specifications. This metric indicates the total number of connections that are used in an instance. The connections can be in the active, idle, or idle in transaction state. The idle in transaction state indicates that a connection is idle in a failed transaction. You can view the connection usage by following the instructions in Manage queries. If connections are insufficient, close idle connections at the earliest opportunity.

Connections by DB(Count)

This metric indicates the number of connections that are used by each database in an instance. When you use this metric, take note of the following items:

• By default, a database supports a maximum of 128 connections. For more information, see Instance specifications.

• If the number of connections used by each database in an instance remains close to the maximum number for a long period of time, a large number of connections exist in the instance. You can check whether the connections are idle or active based on your business scenario. For more information, see Manage connections. Then, release idle connections or scale out the instance at the earliest opportunity based on your business scenario.

• If the number of connections used by specific worker nodes is high and the number of connections used by other worker nodes is low, the connections of the instance are unevenly distributed. You can find and release idle connections at the earliest opportunity to mitigate the uneven connection load. For more information, see Manage connections.

Connections by FE(Count)

This metric indicates the number of connections that are used by each FE node in an instance. When you use this metric, take note of the following items:

• By default, an FE node supports a maximum of 128 connections. For more information, see Instance specifications.

• If the number of connections used by each FE node in an instance remains close to the maximum number for a long period of time, a large number of connections exist in the instance. You can check whether the connections are idle or active based on your business scenario. For more information, see Manage connections. Then, release idle connections or scale out the instance at the earliest opportunity based on your business scenario.

• If the number of connections used by specific worker nodes is high and the number of connections used by other worker nodes is low, the connections of the instance are unevenly distributed. You can find and release idle connections at the earliest opportunity to mitigate the uneven connection load. For more information, see Manage connections.

Max Connection Usage(%)

This metric indicates the highest connection usage among FE nodes. Hologres collects the connection usage of each FE node and displays the highest connection usage by using the Max(frontend_connection_used_rate) function. This allows you to check whether the number of connections exceeds the maximum number supported by an FE node to prevent connection failures. The round robin policy is adopted among multiple FE nodes to establish physical connections to these FE nodes in sequence. If connections are insufficient, you can find and release idle connections. For more information, see Manage connections.

Standard I/O Read Throughput(byte/s)

This metric indicates the amount of data that is read from the standard storage tier in queries per second.

Standard I/O Write Throughput(byte/s)

This metric indicates the amount of data that is written to the standard storage tier in queries per second.

Low Frequency I/O Read Throughput(byte/s)

This metric indicates the amount of data that is read from the IA storage tier in queries per second.

Low Frequency I/O Write Throughput(byte/s)

This metric indicates the amount of data that is written to the IA storage tier in queries per second.

Standard Storage Used(byte)

This metric indicates the amount of standard storage resources that are used to store data. If the purchased storage quota is used up,.you can expand the storage capacity.

Percentage of Standard Storage Used(%)

This metric indicates the percentage of standard storage resources that are used. If the purchased storage quota is used up,.you can expand the storage capacity.

Low Frequency Storage Used(byte)

This metric indicates the amount of IA storage resources that are used to store data. If the purchased storage quota is used up,.you can expand the storage capacity.

Percentage of Low Frequency Storage Used(%)

This metric indicates the percentage of IA storage resources that are used. If the purchased storage quota is used up, you can expand the storage capacity.

FE Replay Running Time(milliseconds)

Hologres is a distributed system. A Hologres instance has multiple FE nodes that parse, distribute, and route SQL statements at the access layer. In Hologres, DDL statements are executed on one FE node and then replayed on other FE nodes. When you use this metric, take note of the following items:

• It is normal if the value of this metric is in milliseconds or seconds.

• If the value of this metric is in minutes, a large number of DDL statements may exist. If the value of this metric continuously increases, queries may be stuck. You can identify and cancel time-consuming queries based on the hg_stat_activity view. For more information, see Manage queries.

• This metric is supported only in Hologres V2.2 and later.

Replica Sync Lag(milliseconds)

This metric indicates the latency of data synchronization between the leader and follower shards after the shard-level replication feature is enabled.

• In most cases, the value of this metric is in milliseconds.

• The value of this metric may increase if a large amount of data is written or updated or DDL statements are frequently executed.

Instance Sync Lag(milliseconds)

This metric indicates the latency of data synchronization from the primary instance to a secondary instance. The latency is displayed in milliseconds. When you use this metric, take note of the following items:

• By default, this metric is not displayed for primary instances and is displayed only for secondary instances.

• This metric is initially displayed as 0 ms after a secondary instance is associated with a primary instance. When data is written to the primary instance, the value of this metric fluctuates.

• In most cases, the synchronization latency between the primary and secondary instances is in milliseconds. The occasional latency jitters are usually caused by metadata modification operations, such as DDL operations, that are performed on the primary instance. These latency jitters can be ignored. If the latency is greater than 1 second over an extended period of time, this can indicate high resource usage. You can check the resource usage such as CPU utilization and memory usage and scale out the instances if necessary to reduce the latency.

• If an instance is restarting or updating, the synchronization latency may increase to several minutes, but will automatically recover after the restart or update is complete.

File Sync Lag(milliseconds)

This metric indicates the latency of file synchronization between geo-disaster recovery instances. This metric is displayed only for read-only secondary instances.

Stats Miss Table Num by DB(countS)

This metric indicates the number of tables for which no statistical information is collected in a database.

• This metric is displayed only in Hologres V2.2 and later.

• In Hologres V2.0 and later, the auto-analyze feature is enabled by default. When a table is newly created or a large amount of data is written or updated, the auto-analyze feature may not update the statistical information of the table in real time. In this case, you can check the table statistics after a period of time.

• If statistical information is not collected for specific tables in a database for several hours or several days, the threshold for triggering the auto-analyze feature may not be reached. In this case, check table statistical information and manually execute the ANALYZE statement. For more information, see ANALYZE and auto-analyze.

Serverless Computing Longest Active Query Time(milliseconds)

Hologres supports the Serverless Computing feature. You can run queries in the serverless computing resource pool. This isolates instance resources and ensures quick query running.

• This metric is displayed only in Hologres V2.1 and later.

• This metric indicates the execution time of the longest-running query that uses serverless computing resources. You can use this metric with the hg_stat_activity table to obtain the running status of queries in the serverless computing resource pool.

Number of data records or bytes consumed in binary logs per second

Hologres allows you to subscribe to Hologres binary logs. You can use binary logs to perform data layering in real time and accelerate data forwarding.

Number of walsenders used and walsender usage

When you use JDBC to consume Hologres binary logs, a walsender is used to connect each shard of the table, which is similar to regular connections. The connections over walsenders are independent of regular connections. A default upper limit is configured for the number of walsenders in an instance.

You can use the two preceding metrics to obtain the walsender usage in an instance. If the upper limit on the number of walsenders is reached, perform troubleshooting by following the instructions in Use JDBC to consume Hologres binary logs.