Application Real-Time Monitoring Service:Trace sampling modes (Agent V3.2.8 or later)

Key concepts

• span: A specific operation in a request, such as a remote call entry or an internal method invocation.

• root span: The first span in a trace.

• local root span: The first span of a trace segment within a single application.

• span context: Information propagated with a request to link all operations within the same trace.

• head-based sampling: A sampling strategy that is performed only at the root span. In most cases, this method preserves a complete trace.

• non-head-based sampling: A sampling strategy that can be triggered at any local root span if head-based sampling does not sample the trace. This method usually cannot guarantee a complete trace.

Sampling strategies and marks

To help you capture more valuable trace data, ARMS provides multiple sampling strategies, including two head-based sampling strategies and three non-head-based sampling strategies.

• Head-based sampling strategies

  • fixed-rate sampling

  • adaptive sampling

• Non-head-based sampling strategies

  • minimum sampling for all interfaces

  • sampling for failed or slow requests

  • custom sampling

Sampling marks

When you use the EagleEye protocol to propagate a trace context across processes, ARMS records whether the trace is sampled. The request header key is EagleEye-Sampled, and its value can be:

• s0: Not sampled

• s1: Sampled

At the local root span where a trace is sampled, the sampling reason is recorded as a span attribute with the key sample.reason. The possible values are:

• s2: minimum sampling for all interfaces

• s3: custom sampling

• s4: fixed-rate sampling

• s5: Reserved

• s6: adaptive sampling

• s7: Reserved

• s8: Basic Edition sampling

• s9: failed request sampling

• s10: slow request sampling

• s11: abnormal call sampling

• s12: business trace sampling

Head-based sampling strategies

ARMS offers two head-based sampling strategies: fixed-rate sampling, the most common approach in distributed tracing, and adaptive sampling, a cost-effective alternative developed by ARMS.

Non-head-based sampling strategies

Head-based sampling strategies cannot guarantee the capture of spans with specific characteristics that are often of high interest, such as spans with errors, high latency, or exceptions, as well as spans from very low-traffic or custom-defined interfaces. The trade-off is that because these strategies make sampling decisions at any point in a trace, they cannot guarantee trace completeness.

Minimum sampling for all interfaces

No configuration is required. This strategy automatically ensures that at least one trace is sampled for each interface per minute. Spans sampled this way are marked with the attribute sample.reason and the value s2.

Sampling for failed or slow requests

A trace is sampled if an interface call meets any of the following conditions:

• Interface error: For HTTP-based interfaces, the response code is not 200. For other types of interfaces, the instrumented method throws an exception.

• Internal execution exception: An exception occurs during the internal execution of an interface call but is not propagated to the framework's entry point instrumentation.

• High latency: The call duration of an interface exceeds the interface slow-call threshold configured on the Custom Configurations page.

  Note

  If quantile statistics are also enabled, the slow request sampling rule is also triggered if the call duration exceeds the P99 latency for that interface.

Spans sampled for these reasons are marked with the sample.reason attribute, which is set to s9 (interface error), s11 (internal exception), or s10 (high latency), respectively.

Custom sampling

If other sampling strategies do not capture all the traces you need, you can use custom sampling to ensure that specific interfaces are always sampled (100% sampling rate). You can define interfaces by their exact name, a prefix, or a suffix. A span sampled by this strategy carries an attribute with the key sample.reason and the value s3.

To configure custom sampling:

1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring > Application List.

2. At the top of the Application List page, select the destination region, and then click the target application.

   Note

   The icons in the Language column have the following meanings:

   : A Java application connected to Application Monitoring.

   : A Go application monitored by Application Monitoring.

   : A Python application monitored by Application Monitoring.

   -: An application integrated with Managed Service for OpenTelemetry.

3. In the top navigation bar, choose Application Configuration > Custom Configurations.

4. In the Sampling Settings section, specify the interfaces, interface prefixes, and interface suffixes that you want to be fully sampled.

   Note

   Changes take effect immediately without requiring an application restart.

5. Click Save.

How it works

Consider a trace path of A > B > C. The final decision to report spans for a business call is determined by a combination of the sampling strategies. The decision process, which is shown in the following flowchart, runs each time a request arrives at A, B, and C. However, some steps are skipped depending on whether the current span is a root span or a local root span.

The colors in the flowchart have the following meanings:

• Purple: Standard head-based sampling. It is triggered only at the root span of the trace. In the A > B > C example, this strategy is triggered only at A.

• Blue: This strategy can be triggered at any node in the trace if head-based sampling is not triggered. For example, if A decides not to sample a request, B re-evaluates whether to sample it based on its own custom sampling and minimum sampling for all interfaces strategies. If B decides to sample, it propagates the sampling mark to C. This strategy can be triggered at A, B, and C.

• Green: This strategy can be triggered at any node if the current sampling decision is "do not sample," but it does not affect the sampling decision for downstream services. For example, if A decides not to sample a request, B might still decide to sample it based on the sampling for failed or slow requests strategy. In this case, B does not propagate the sampling mark to C. This strategy can be triggered at A, B, and C.

Related documents

After traces are collected, you can analyze the stored data in real time by combining various filter conditions and aggregation dimensions. For more information, see trace analysis.