Application Real-Time Monitoring Service:Trace sampling modes for agents before 3.2.8

In most distributed systems, not every trace is worth recording. Many traces contain repetitive or low-priority information. Sampling helps reduce observability costs and filter out unhelpful noise.

The fundamental principle of trace sampling is to prioritize recording the traces that are most important and most likely to be accessed. ARMS provides the following sampling modes:

• Fixed-rate sampling

• Adaptive sampling

• Basic Edition sampling

• Sampling for failed or slow requests

Fixed-rate sampling

Fixed-rate sampling records a specific percentage of traces based on the TraceId. For example, a rate of 10% means one out of every 10 traces is recorded. This method ensures that an entire trace is either fully kept or fully discarded, preventing incomplete trace data.

To configure fixed-rate sampling, follow these steps:

1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring > Application List.

2. At the top of the Application List page, select the destination region, and then click the target application.

   Note

   The icons in the Language column have the following meanings:

   : A Java application connected to Application Monitoring.

   : A Go application monitored by Application Monitoring.

   : A Python application monitored by Application Monitoring.

   -: An application integrated with Managed Service for OpenTelemetry.

3. In the left-side navigation pane, click Application Settings. On the page that appears, click the Custom Configuration tab.

4. In the sampling rate settings section, set the sampling rate. Set sampling mode to fixed-rate sampling, and in the sampling rate field, enter the percentage value. For example, enter 10 for 10%.

   Note

   The changes take effect immediately without requiring an application restart. The default sampling rate differs by language: for Java and Go applications, the default is 10; for Python applications, the default is 100. A higher rate consumes more system resources. We recommend keeping the default value.

Adaptive sampling

To further reduce costs and improve trace query performance, ARMS provides adaptive sampling. Unlike fixed-rate sampling, adaptive sampling uses multiple policies to dynamically decide whether to sample a trace. This approach addresses the coverage issues of low-rate sampling and the cost issues of high-rate sampling. We recommend adaptive sampling for high-traffic scenarios or for applications with fluctuating traffic patterns.

Adaptive sampling uses three main policies: full sampling for specific APIs, Top-N sampling for APIs, and minimum sampling for low-traffic APIs.

• Full sampling for specific APIs: You can specify API names, prefixes, or suffixes to mark APIs that require 100% sampling. All requests to these APIs will trigger 100% sampling for the entire trace. Enable this only for your most critical APIs or for temporary debugging, as it significantly increases the amount of data collected.

• Top-N sampling for APIs: This uses a modified LFU (Least Frequently Used) algorithm to sample a limited number of requests for each API within a given period. This ensures that data collection does not grow linearly with API traffic.

• Minimum sampling for low-traffic APIs: This guarantees at least one trace is sampled for each API within a given period. This ensures that valuable characteristic information is recorded for every business API, even during periods of low traffic.

To configure adaptive sampling, follow these steps:

1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring > Application List.

2. At the top of the Application List page, select the destination region, and then click the target application.

   Note

   The icons in the Language column have the following meanings:

   : A Java application connected to Application Monitoring.

   : A Go application monitored by Application Monitoring.

   : A Python application monitored by Application Monitoring.

   -: An application integrated with Managed Service for OpenTelemetry.

3. In the left-side navigation pane, click Application Settings. On the page that appears, click the Custom Configuration tab.

4. In the sampling rate settings section, set the sampling rate. Set sampling mode to adaptive sampling. You can then specify the names, prefixes, and suffixes of the APIs that you want to fully sample.

   Note

   The changes take effect immediately without requiring an application restart. Adaptive sampling requires agent version 2.8.3 or later.

Basic Edition sampling

Basic Edition sampling is available only to users of Application Monitoring Basic Edition. It includes one free sampling policy and multiple custom sampling policies.

• Default sampling (Free): By default, ARMS collects one trace per minute from each agent at no cost for all APIs under your account.

• Custom sampling policy: You can configure custom sampling policies. Each policy can sample by a fixed percentage or a fixed traffic quota, and can be applied to all or specific APIs.

To configure a custom sampling policy, follow these steps:

1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring > Application List.

2. At the top of the Application List page, select the destination region, and then click the target application.

   Note

   The icons in the Language column have the following meanings:

   : A Java application connected to Application Monitoring.

   : A Go application monitored by Application Monitoring.

   : A Python application monitored by Application Monitoring.

   -: An application integrated with Managed Service for OpenTelemetry.

3. In the left-side navigation pane, click Application Settings. On the page that appears, click the Custom Configuration tab.

4. In the sampling rate settings section, click Add Client Sampling Policy, set the following parameters, and then click OK.

   Parameter	Description
   Policy Name	A custom name for the sampling policy.
   Sampling Type and Samples	Fixed-rate Sampling: Samples traces based on a fixed proportion. If you select this option, set the Samples to a fixed percentage, such as 10%. Traffic Limit: If you select this option, set the Samples to the number of traces each agent collects within the specified time window. For example, a setting of 5 traces per agent per 1 second means that each agent collects up to 5 traces every second.
   Applicable interface	Specify the scope to which the sampling policy applies. You can select per span or select Specify a span and enter the interface name. Note Currently, each sampling policy supports only one interface name. To sample traces for multiple interfaces, you must configure multiple sampling policies.

Examples of custom sampling policies:

• For the /elastic/update API, sample 20 traces per minute.

• For all APIs, sample 20% of the traces.

Sampling for failed or slow requests

This mode samples all related spans for a request if it meets the following criteria:

• The request fails. A failure is determined by one of two conditions:

  • For HTTP APIs, the status code is not 2xx or 3xx.

  • An exception is caught by the LocalRootSpan instrumentation method, which typically occurs when a business exception is thrown to the framework.

• The duration of the call exceeds the P99 latency of previous calls to the same API. Note: This P99 value is calculated using bucketing and is therefore an approximation.

• An internal method within the call throws an exception (supported in agent version 4.1.x and later).

This sampling policy does not guarantee a complete end-to-end trace. It only saves spans related to the call within a single application.

Additional information

To prevent performance degradation during high-traffic periods, ARMS limits the number of traces an agent can collect per second. The default is 100 traces. This safeguard applies to both fixed-rate and adaptive sampling. You can change this limit by modifying the throttling threshold on the custom configuration tab.

Related documents

After traces are collected, you can analyze the stored data in real time by combining various filter conditions and aggregation dimensions. For more information, see trace analysis.