Slow requests are a common cause of connection timeouts that affect the service quality of Tair (Redis OSS-compatible). The Tair (Redis OSS-compatible) slow query log system records each slow request and the client IP address that sent it, so you can identify both the request and its source. Follow the procedure in this topic to locate the cause of a timeout, and use the alternative paths when slow query logs do not explain the issue.
Viewing methods
| Type | Method |
| Data node slow query log | Connect to the instance from a client and run the SLOWLOG GET command. For details, see SLOWLOG GET. Use the console or call an API operation: Query slow query logsDescribeSlowLogRecords |
| Proxy node slow query log | Use the console or call an API operation: Query slow query logsDescribeSlowLogRecords |
Choose a troubleshooting path
Slow query logs are the primary method for troubleshooting timeout issues, but they do not cover every cause. Match what you observe to the corresponding path in this topic.
| Observable condition | Path |
| A service timeout occurs | Start with the proxy node slow query log, then check the data node slow query log. See Analyze slow query logs to identify the timeout cause. |
| A service timeout occurs and the instance uses the standard architecture | Skip the proxy node slow query log and analyze the data node slow query log. See Analyze slow query logs to identify the timeout cause. |
| The proxy node slow query log is empty and the instance is accessed over a public network (Internet) | Check the network link between the client and the instance. See Troubleshoot public network connection latency. |
Read requests on a key occur at a high frequency but each completes faster than the slowlog-log-slower-than threshold, so they are not recorded in the slow query log | See Slow query log and audit log limits for hot key troubleshooting. |
| The slow query log analysis and the public network connection check do not identify the cause | See Continue troubleshooting on the VPC, ECS, and application side. |
Check server-side performance first
The slow query log records only requests that exceed the slow query threshold. Before you decide whether a timeout comes from a slow request or from the network link, confirm whether the instance itself has a performance bottleneck.
Run AI Instance Inspection
On the product page, click AI Instance Inspection.
Select all eight inspection items and click Start Inspection.
Review the generated inspection report for anomalies in instance status, security, high availability (HA), data node performance, proxy node performance, slow query logs, big key and hot key conditions, and event alerting.
View performance monitoring
On the product page, click Performance Monitoring.
Switch to the Data Node view or Proxy Node view, and review CPU usage, requests per second (QPS), average response time (RT), and inbound and outbound traffic rates.
Use the metrics for each node to determine whether the instance has a performance issue.
Analyze slow query logs to identify the timeout cause
Service timeouts can have complex causes, but they are often related to slow requests. Follow these steps to troubleshoot timeout issues.
When a service timeout occurs, first check the proxy node slow query log. For instructions, see Query slow query logs.
NoteIf your instance uses the standard architecture, skip to Step 3 to analyze the data node slow query log.
If the proxy node slow query log is empty, check the network connection between the client and the instance. If the instance is accessed over a public network (Internet), see Troubleshoot public network connection latency below.
Identify the command from the earliest entry in the proxy node slow query log.
NoteProxy node slow query logs are typically generated when slow requests on a data node cause commands to pile up.
In this example, the earliest slow query log entry is generated by a KEYS command. The IP address in the log record is the client IP address that sent the command.
On the Slow Logs page, click the Proxy Node tab to view the proxy node slow query log. In this example, the table displays five records: one for a
KEYScommand and four forSETcommands, with execution times ranging from 64,048 to 88,861 microseconds.Check the data node slow query log to see if the command found in the proxy log also appears, which can confirm it as the root cause.
NoteTypically, the command that first generates a slow query log entry on a proxy node also generates an entry on a data node. The data node slow query log usually has fewer entries than the proxy node slow query log because the two logs define execution time differently and use different slow query thresholds.
In this example, comparing the two logs reveals that an entry for the KEYS command exists in both. The absence of other commands from the proxy log in the data node log confirms that the KEYS command is the root cause.
On the Slow Logs page, click the Data Node tab to view the data node slow query log. In this example, only a slow query log entry for the
KEYScommand appears. Its long execution time confirms that this command is the root cause of the timeout.In the proxy node slow query log, search for the command from the previous step to find the source client IP address. Then, optimize the client application.
If the slow query log analysis and the public network connection check do not identify the cause, see Continue troubleshooting on the VPC, ECS, and application side.
Slow query log and audit log limits for hot key troubleshooting
The slowlog-log-slower-than parameter sets the threshold for the data node slow query log. The default value is 20,000 microseconds (20 ms). Read requests that occur at a high frequency but individually complete faster than this threshold are not recorded. As a result, the default slow query log cannot capture the read requests that cause a hot key issue. If you suspect a hot key issue, lower the value of slowlog-log-slower-than to log more requests and identify the source of the high-frequency read requests. For instructions, see Query slow query logs.
After you enable audit logs, only write operations are recorded. Read operations are not recorded. As a result, you cannot use audit logs to view the client IP addresses of read requests that access a hot key. To locate these client IP addresses, lower the slowlog-log-slower-than threshold and check the resulting slow query logs. For more information about the audit log feature, see Overview of audit logs. If the hot key issue is caused by write requests instead, use the audit log-based hot key query feature. For instructions, see Query historical hotkeys.
Troubleshoot public network connection latency
If server-side metrics show no performance bottlenecks (CPU, memory, and connections are all normal, and the slow query log is empty), but the client still reports connection timeout errors, check the network link between the client and the instance.
Common symptoms
The client reports errors such as socket error, read error, or connection timeout, while the instance server-side CPU, memory, and connection metrics are normal, and RT is low.
Main cause
The public network link between the client and the instance is unstable, and cross-region transmission adds delay. Both effects are more pronounced in cross-border access scenarios.
Recommendations
Use an internal network connection for core business workloads instead of relying on a public network link.
For cross-region deployments, use Cloud Enterprise Network (CEN) to connect VPCs across regions and achieve low-latency, stable internal network connectivity. For instructions, see Manage inter-region connections.
On the client side, increase the connection timeout and read/write timeout values, and implement retry logic to handle network fluctuations. For common timeout error solutions, see Common errors and troubleshooting.
Continue troubleshooting on the VPC, ECS, and application side
If the slow query log analysis and the public network connection check do not identify the cause, use VPC, ECS, and application-side information to narrow the scope.
Obtain the VPC ID and vSwitch ID from the product page.
Check whether the ECS instance experiencing slowness or timeouts is abnormal, and whether the timing coincides with an application change.
If the timing coincides with an application change, roll back the change or investigate it for clues. Use the VPC ID and vSwitch ID to continue troubleshooting in the VPC or ECS console.