This topic describes a variety of trace sampling policies that are supported by Application Real-Time Monitoring Service (ARMS). You can select appropriate trace sampling policies based on your scenarios so that you can obtain the trace data that you want at a low cost.

Trace sampling is suitable for high-traffic applications that have a large number of visits. Trace sampling can help you record the most valuable trace data at a low cost and a low performance overhead. The basic principle of trace sampling is to preferentially record the traces that you are most concerned about and most likely to access. ARMS provides the following trace sampling policies:

Note You can use the preceding trace sampling policies in combination to fully meet your personalized sampling requirements.

Trace feature-based sampling

Trace feature-based sampling refers to the sampling based on the attributes of traces, such as time consumption and status. ARMS supports fixed-rate sampling and sampling based on thread profiling for slow calls.

Fixed-rate sampling

Fixed-rate sampling records a specific proportion of trace data based on the ordinal number of TraceId. For example, if the fixed rate is 10%, one out of every 10 pieces of trace data is recorded. Fixed-rate sampling avoids incomplete trace data. The data of an entire trace is retained or discarded.

Fixed-rate sampling applies to the following scenarios:

  • During the peak hours of stress testing or big promotions, the traffic volume is high. If full trace logs are reported, the client performance may be degraded. To avoid this situation, we recommend that you reduce the fixed sample rate to a value between 1% and 10%.
  • On regular days, the cost of network bandwidth is high because full trace logs are reported. In this case, you can consider adjusting the fixed sample rate as needed.

The following figure shows how fixed-rate sampling works.

dg_part

You can perform the following steps to configure fixed-rate sampling:

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Application Monitoring > Applications and select a region in the top navigation bar.
  3. On the Applications page, click the name of the application.
  4. In the left-side navigation pane, click Application Settings. On the page that appears, click the Custom Configuration tab.
  5. In the Invocation Trace Sampling Settings section, turn on or turn off the switch of trace sampling and specify a sample rate. In the Sampling Rate Settings field, enter the number of the percentile. For example, if you enter 10, the sample rate is 10%.
    Notice The modification takes effect immediately. You do not need to restart the application. If sampling is turned off, the trace data is not captured. Proceed with caution.

Sampling based on thread profiling for slow calls

Sampling based on thread profiling for slow calls records data of traces in which thread profiling is triggered to listen on slow calls. After thread profiling is enabled, ARMS transfers a sampling mark to the downstream at the same time when ARMS records a slow call. ARMS also retains the downstream traces of this slow call. However, the number of listening threads of thread profiling is limited to avoid compromised client performance. If multiple slow calls occur at the same time, only some slow calls that meet the conditions and their downstream traces are recorded.

Sampling based on thread profiling for slow calls applies to the following scenarios:

  • The system encounters occasional slow calls. For example, the response time of services surges from 0.5s to 10s at night. Assume that you enable thread profiling in advance (the default trigger threshold is 2s). ARMS automatically records the native method stacks of slow calls that are complete within 2s to 10s in the request and the downstream traces of the slow calls.
  • The system responds at a slow speed during the peak hours of flash sales or periodic big promotions. In this case, thread profiling records the native method stacks of slow calls that reach the trigger threshold and the downstream traces of the slow calls.

The following figure shows how sampling based on thread profiling for slow calls works.

dg_slow

You can perform the following steps to configure sampling based on thread profiling for slow calls:

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Application Monitoring > Applications and select a region in the top navigation bar.
  3. On the Applications page, click the name of the application.
  4. In the left-side navigation pane, click Application Settings. On the page that appears, click the Custom Configuration tab.
  5. In the Thread settings section, turn on or turn off the switch of the thread diagnostics method stack and the master switch of thread profiling. Specify a threshold to trigger slow call listening.
    Thread Settings
    Note The listener is started only when the service call response time exceeds the threshold (1,000 ms by default) and lasts until the call ends or the consumed time exceeds 15 seconds. We recommend that you set the threshold to the 99th percentile of the call response time. For example, if 100 calls are listed in ascending order by response time, the time consumed by the 99th one is the 99th percentile.

Business feature-based sampling

Business feature-based sampling refers to the sampling based on the business traffic features of applications. ARMS allows you to configure rules based on the HTTP traffic features of entry applications. You can filter Header, Method, Cookie, and Parameter information when you extract business features so that the conditions for matching business features can be met in various scenarios. After the switch of full collection is turned on, the trace data that meets the business conditions is given priority to be fully collected. The configuration immediately takes effect, and can be dynamically modified during runtime.

Business feature-based sampling applies to the following scenarios:

The following figure shows how business feature-based sampling works.

dg_scenario

You can perform the following steps to configure sampling based on business features:

  1. Log on to the ARMS console.
  2. In the Basic information section on the New business monitoring page, set related parameters. Click Advanced Settings. In the Advanced Settings section, set related parameters. Then, click Save.
    pg_business
    Parameter Description Example
    Business Name Required. The name of the business transaction task. Business feature test
    Entry application Required. The Entry application drop-down list displays all the Java applications that have the ARMS agent installed. After you select the required application, ARMS automatically detects the version number of the agent.
    Note You can use the business transaction feature only after you upgrade the ARMS agent to a version later than 2.6.2. If the detected agent version is not later than 2.6.2, upgrade the agent first. For more information, see Update the ARMS agent for Java applications.
    		mall-center
    Service type Required. The type of the service. ARMS supports only HTTP entry. This service type is suitable for scenarios where business traces are dyed based on HTTP traffic features. HTTP entry
    Service name Required. The API name that is provided by the application. ARMS automatically detects a list of APIs that are recently provided by this application based on the entry application you specify for your choices. If the recommended APIs do not meet your requirements, you can edit the APIs.
    The following four types of matching patterns of Service name are supported:
    • Equal: matches the business API that uses the value of the Service name parameter as the API name. This pattern is the default matching pattern.
    • Start equal: matches the business APIs whose names are prefixed with the value of the Service name parameter. If you need to monitor the business APIs that have the same prefix, select this pattern.
    • Contains: matches the business APIs whose names contain the value of the Service name parameter. If your application provides a large number of business APIs, you can select this pattern to quickly monitor the business APIs that you need.
    • End =: matches the business APIs whose names are suffixed with the value of the Service name parameter. This pattern is suitable for typical web frameworks that end with the .do and .action configurations.
    • Pattern matching: matches dynamic URI paths and supports matching rules for Ant-style path patterns. This allows you to monitor and analyze the URIs of a specific type of patterns.
    		Equal /api/buy
    Filter rule relationships You can select Meet the following rules at the same time or Meet the following rule. Meet the following rules at the same time
    Filter rules Optional. Further filter the business APIs that you specify. To specify Filter rules, you must specify a parameter to be matched (Parameter, Cookie, Method, PathVariable, or Header), a key value to be matched, a matching pattern (==, !=, or contains), and a threshold. Among the preceding information, the PathVariable option appears for the parameter to be matched only if you select Pattern matching for the Service name parameter and the input string includes braces {} as placeholders. You can set multiple filter rules. The logical relationship between multiple filter rules is determined by the Filter rule relationships parameter that you specify. Assume that your application provides the /api/buy?brand=*** URL and you want to monitor the API calls of brand=Alibaba. You can set the Filter rules parameter to Parameter brand == Alibaba in the form.
    Whether full collection Specifies whether to collect all the traces that have dye labels. Yes

O&M feature-based sampling

O&M feature-based sampling refers to the sampling based on the O&M features of applications, such as the deployment environment and the network environment. ARMS supports sampling based on the Kubernetes Pod Metadata features of applications. ARMS allows you to configure O&M feature rules based on the Kubernetes Pod Metadata features of entry applications. ARMS can automatically identify the metadata information of pod instances of applications, such as the label, annotation, pod name, and namespace, and filter the metadata information. This meets the conditions for matching O&M features in various scenarios. After the switch of full collection is turned on, the trace data that meets the O&M feature conditions is given priority to be fully collected. The configuration immediately takes effect, and can be dynamically modified during runtime.

O&M feature-based sampling applies to the following scenarios:

  • Specific sampling settings are required based on the difference of information about application agents, such as languages, versions, environment variables, and startup parameters.
  • Specific sampling settings are required based on the difference of information about servers where applications are deployed, such as specifications, models, regions, and zones.
  • Specific sampling settings are required based on the difference of application deployment environments, such as the development, testing, pre-release, and production isolation requirements.
  • Specific sampling settings are required based on the difference between deployment modes of network environments where applications reside, such as physical machines, virtual machines, and containers. Alternatively, specific sampling settings are required based on the difference between the classic network and a VPC.

The following figure shows how O&M feature-based sampling works.

dg_k8s_ts

You can perform the following steps to configure O&M feature-based sampling:

  1. Log on to the ARMS console.
  2. In the Basic information section on the New business monitoring page, set related parameters. Click Advanced Settings. In the Advanced Settings section, set related parameters. Then, click Save.
    pg_k8s
    Parameter Description Example
    Business Name Required. The name of the business transaction task. O&M feature test
    Entry application Required. The Entry application drop-down list displays all the Java applications that have the ARMS agent installed. After you select the required application, ARMS automatically detects the version number of the agent.
    Note You can use the business transaction feature only after you upgrade the ARMS agent to a version later than 2.6.2. If the detected agent version is not later than 2.6.2, upgrade the agent first. For more information, see Update the ARMS agent for Java applications.
    		product-center
    Service type Required. The type of the service. ARMS supports only Kubernetes Pod Metadata. This service type is suitable for scenarios where business traces are dyed based on the environment features of Kubernetes pods. Kubernetes Pod Metadata.
    Filter rule relationships You can select Meet the following rules at the same time or Meet the following rule. Meet the following rules at the same time
    Filter rules Optional. Further filter the business APIs that you specify. To specify Filter rules, you must specify a parameter to be matched (podLabel, podAnnotation, podName, podNamespace, podUID, podIp, nodeName, hostIp, or podServiceAccount), a matching pattern (==, !=, or contains), and a threshold. You can set multiple filter rules. The logical relationship between multiple filter rules is determined by the Filter rule relationships parameter that you specify. Assume that your application is deployed in the dev, test, staging, and prod environments. The Pod label stage is used for environment differentiation. You want to monitor the dev environment and collect all traces. In this case, you can set the Filter rules parameter to podLabel stage == dev in the form.
    Whether full collection Specifies whether to collect all the traces that have dye labels. Yes

Time feature-based sampling

Time feature-based sampling refers to the sampling based on the time features of applications, such as online diagnostics and offline analysis. Requirements on sample rates vary based on diagnostic scenarios. ARMS supports temporary full collection in online diagnostic scenarios and helps you quickly locate online problems. After real-time diagnostics is enabled, ARMS continuously monitors the application for 5 minutes and reports all the trace data during this period. Then, you can start from the trace that has performance problems, and use features, such as the waterfall charts of method stacks and thread profiling, to identify the causes of the problems.

Time feature-based sampling applies to the following scenarios:

You need to closely monitor the application performance for a short period of time, for example, when you release an application or perform stress testing on the application.

The following figure shows how time feature-based sampling works.

dg_temp

You can perform the following steps to configure time feature-based sampling:

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Application Monitoring > Applications and select a region in the top navigation bar.
  3. On the Applications page, click the name of the application.
  4. In the left-side navigation pane, choose Application Diagnosis > Real-time Diagnosis.

    The first time you go to the Real-time Diagnosis page, real-time diagnostics is automatically enabled. To enable real-time diagnostics in other cases, click Activate Real-time Diagnosis in the upper-right corner of the page.

    Real-time diagnostics is automatically enabled for 5 minutes and then disabled. To disable real-time diagnostics before it is automatically disabled, click Pause Auto-Refresh in the upper-right corner.