If your application encounters issues such as uneven traffic, instance failures, slow endpoints, business traffic analysis, or canary release monitoring, you can use Trace Explorer in application monitoring to quickly pinpoint problematic code. This topic uses five classic production issues to demonstrate how to use Trace Explorer and the value it provides.
Background information
Besides using a trace to investigate an anomaly in a single request or using pre-aggregated metrics for service monitoring and alerting, distributed tracing also supports post-aggregation analysis on detailed trace data. This feature is known as Trace Explorer. Compared to analyzing individual traces, Trace Explorer helps you identify issues faster. Compared to pre-aggregated monitoring charts, Trace Explorer offers more flexibility for custom diagnostics.
Trace Explorer performs real-time analysis on all stored, detailed trace data. You can freely combine filter conditions and aggregation dimensions for custom diagnostics in various scenarios. For example, you can view the time-series distribution of calls that take longer than 3 seconds, check how error requests are distributed across different machines, or monitor traffic changes for VIP customers.
Issue 1: Uneven traffic distribution
What do you do when a load balancing misconfiguration routes a high volume of requests to a few machines, creating "hotspots" that impact service availability?
"Hotspots" caused by uneven traffic distribution can easily lead to service unavailability. Many such cases occur in production environments due to factors like load balancing misconfigurations, service registry anomalies preventing restarted nodes from coming online, or abnormal DHT hash factors.
The biggest risk with uneven traffic is the failure to detect a "hotspot" in time. Its symptoms often manifest as slow responses or errors, and traditional monitoring may not directly show the traffic imbalance. As a result, operations personnel might not consider it the primary cause, wasting valuable incident response time and letting the problem worsen.
With Trace Explorer, you can group trace data by IP address to visualize how requests are distributed across your machines, especially the change in traffic distribution before and after an incident. If a large number of requests suddenly become concentrated on one or a few machines, it is likely a hotspot caused by uneven traffic. You can then correlate this with change events at the time of the incident to quickly identify the faulty change and perform a rollback.
On the Trace Explorer page, if you aggregate by IP address, you might discover that most traffic is concentrated on the opentelemetry-demo-frontend-XX machine. On the Trace Explorer page, set the aggregation dimension to IP address and sort by request count to view the request count, error count, and average duration for each host. For example, the opentelemetry-demo-frontend host has a request count as high as 17,000, far exceeding other hosts, which indicates a clear traffic concentration.
Issue 2: Instance failure
How do you investigate when an instance failure, such as a damaged network card, CPU overcommitment, or a full disk, causes some requests to fail or time out?
Instance failures occur frequently, especially in large core clusters where, statistically, they are almost inevitable. An instance failure might not cause a large-scale outage, but it can lead to a small number of user requests failing or timing out. This continuously degrades the user experience and increases support costs, so it must be addressed promptly.
Instance failures can be categorized into host failures and container failures (or Node and Pod failures in a Kubernetes environment). For example, CPU overcommitment and hardware faults are host-level issues that affect all containers on the host. In contrast, issues like a full disk or an out-of-memory error typically affect only a single container. Therefore, when troubleshooting instance failures, you can analyze data from both the host IP and container IP dimensions.
To address this problem, use Trace Explorer to first filter for abnormal or timed-out requests. Then, aggregate the results by host IP or container IP address to quickly determine if an instance failure is the cause. If abnormal requests are concentrated on a single machine, you can try replacing the machine for a quick recovery or inspect its system metrics, such as whether the disk is full or if CPU steal time is too high. If the errors are scattered across multiple machines, you can likely rule out an instance failure and should focus on analyzing downstream dependencies or application logic for issues.
On the Trace Explorer page, filter for error or slow calls and group the results by IP address. If abnormal calls are concentrated on a specific machine, an instance failure is highly probable. On the Trace Explorer page, set the filter condition to statusCode IN (2, 3), select IP address as the aggregation dimension, and check the Error status in the quick filter section on the left. The query results show a total of 2,857 error calls, with two hosts (*.42 and *.47) generating 1,430 and 1,425 calls respectively, both with a 100% error rate. The errors are concentrated between 19:21 and 19:26, which is characteristic of a typical instance failure.
Issue 3: Manage slow endpoints
How do you quickly identify slow endpoints and resolve performance bottlenecks before a new application launch or a major sales promotion?
Systematic performance tuning is often necessary when launching a new application or preparing for a major promotion. The first step is to analyze the current system for performance bottlenecks by identifying a list of slow endpoints and how often they occur.
You can use Trace Explorer to filter for calls with a duration exceeding a certain threshold and then group them by endpoint name. This allows you to quickly identify the list of slow endpoints and their patterns. You can then address the most frequently occurring slow endpoints one by one.
After finding the slow endpoints, you can use the corresponding traces, method stacks, and thread pool data to locate the root cause of the slow calls. Common causes include:
-
The database or microservice connection pool is too small, causing many requests to wait for a connection. This can be resolved by increasing the maximum thread count of the connection pool.
-
The N+1 query problem. For example, a single external request triggers hundreds of internal database calls. You can merge these fragmented requests to reduce network transfer time.
-
The data size of a single request is too large, leading to long network transfer and deserialization times, and can easily cause a Full GC. You can switch from a full query to a paginated query to avoid requesting too much data at once.
-
A "hot lock" in the logging framework. You can switch from synchronous to asynchronous log output.
On the Trace Explorer page, filter for slow calls that take longer than 5 seconds and group them by endpoint name to identify patterns among slow endpoints.
Issue 4: Business traffic analysis
How do you analyze traffic changes and service quality for key customers or channels?
In a production environment, services are often standardized, but business segments are tiered and classified. For the same order service, we need to categorize and aggregate statistics by dimensions like category, channel, and user for fine-grained operations. For example, in the offline retail channel, the stability of every order and every POS terminal can become a public relations issue. The SLA requirements for offline channels are much higher than for online channels. So, how can you accurately monitor the traffic status and service quality of the offline retail trace within a generic e-commerce service system?
You can use Trace Explorer's filtering and statistics on custom attributes to perform low-cost, business-oriented trace analysis. For example, you can add a tag like {"attributes.channel": "offline"} to traces for offline orders at the entry-point service, and then add separate tags for different stores, customer groups, and product categories. Finally, by filtering for attributes.channel = offline and using group by on different business tags to aggregate metrics like request count, duration, or error rate, you can quickly analyze the traffic trends and service quality for each business scenario.
Issue 5: Canary release monitoring
You are deploying to 500 machines in 10 batches. How can you quickly determine if there are any issues after the first canary release batch goes live?
The three pillars of change management—"canary, monitor, rollback"—are essential for online stability. Phased canary releases are a key method for reducing online risks and controlling the blast radius. If an anomaly is detected in a canary batch, it should be rolled back immediately instead of continuing the deployment. However, many production failures occur due to a lack of effective canary monitoring.
For example, if a microservice's service registry is unavailable, restarted machines cannot register or come online. Without canary monitoring, the first few batches of restarted machines might all fail to register. All traffic is then routed to the remaining active machines. The overall application traffic and duration may not change significantly until the last batch of machines also fails to register, at which point the entire application becomes unavailable, leading to a major production incident.
In the scenario above, if you tag traffic from different machine versions with {"attributes.version": "v1.0.x"}, you can use Trace Explorer to group statistics by attributes.version. This allows you to clearly distinguish traffic changes and service quality between different versions or before and after a release, preventing a canary batch anomaly from being masked by global monitoring metrics.
Limitations of Trace Explorer
While Trace Explorer is flexible and can meet various custom diagnostic needs, it has a few limitations:
-
Analysis of detailed trace data is costly.
Trace Explorer requires you to report and store detailed trace data as completely as possible. If the sampling rate is low and the detailed data is incomplete, the effectiveness of Trace Explorer is greatly reduced. To lower the cost of full storage, you can deploy edge data nodes within your clusters for temporary data caching and processing to reduce cross-network reporting overhead. Alternatively, you can implement hot and cold data separation on the server side: perform full trace analysis on hot storage, and diagnose only slow or erroneous traces on cold storage.
-
Post-aggregation is not suitable for alerting due to high query overhead and low concurrency.
Trace Explorer performs real-time scans and statistics on full datasets. The query performance overhead is much higher than with pre-aggregated metrics, making it unsuitable for high-concurrency alert queries. Instead, use the custom metrics feature to push down post-aggregation statements to the client for custom metric collection, which can then be used for alerting and custom dashboards.
-
Custom attribute tagging is required to maximize the value of Trace Explorer.
Unlike standard pre-aggregated metrics in application monitoring, many scenarios in Trace Explorer require you to manually instrument your code with custom tags. This is the most effective way to differentiate between various business scenarios and enable precise analysis.
Related documents
To diagnose problems proactively, you can use the alert feature of Application Real-Time Monitoring Service (ARMS) to create an alert for a specific endpoint or for all endpoints. When an issue occurs, the service sends a notification to your operations team. To learn how to create an alert, see Application Monitoring alert rules.