Application Real-Time Monitoring Service:Customize settings for a Java application

Prerequisite

An ARMS agent is installed for the application.

Feature access

1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring > Application List.

2. On the Applications page, select a destination region at the top of the page, and then click the name of the application.

   Note

   The icons in the Language column have the following meanings:

   : a Java application connected to Application Monitoring.

   : a Go application connected to Application Monitoring.

   : a Python application connected to Application Monitoring.

   -: an application connected to Managed Service for OpenTelemetry.

3. In the top navigation bar, choose Configuration > Custom Configurations.

4. Set the custom configuration parameters and click Save.

Sampling settings

In the Sampling Settings section, you can configure a sampling policy, sample rate, and features for full sampling of interfaces. You can also enable or disable low-traffic sampling. For more information, see Select a trace sampling mode for an agent of V3.2.8 or later.

Agent switch settings

In the Agent switch settings section, you can enable or disable application monitoring and select the plug-ins that you want to use.

Thread profiling settings

In the Thread Profiling Settings section, you can enable or disable the main switch for thread profiling.

Application log association settings

In the Application Log Association Settings section, you can configure a log source to associate with the application and specify whether to automatically inject TraceId and SpanId into logs. For more information, see Log analysis.

• Auto Fill TraceId and SpanId: If you enable this feature, Trace ID and Span ID are automatically injected into logs. You do not need to manually modify the log configuration file.

  Effect:

  As shown in the following figure, log4j, log4j2, and logback automatically print the Trace ID and Span ID in logs without any modification to the log configuration file.

  

• Support adding traceId in log MDC: By default, agents of V4.x do not support obtaining TraceId using the org.slf4j.MDC.get(“EagleEye-TraceID”) method. If you still need to use this method, you can enable this switch.

URL convergence settings

In the URL Convergence Settings section, you can enable or disable convergence, and specify a convergence threshold and convergence rules. URL convergence displays similar URLs as a single object. For example, URLs prefixed with /service/demo/ are displayed as an object. The convergence threshold is the minimum number of URLs that can trigger URL convergence. For example, if the threshold is set to 100, URL convergence is triggered only when at least 100 URLs match the regular expression of a rule. For more information, see ARMS convergence policies.

Arthas monitoring

In the Arthas monitoring section, you can enable or disable the Arthas diagnostics feature. You can also specify the IP addresses on which you want to perform Arthas diagnostics. For more information, see Arthas diagnostics.

Continuous profiling settings

In the Continuous Profiling Settings section, you can enable or disable the main switch and the switches for CPU Hotspot, Memory Hotspot, and Code Hotspot. You can also specify the IP addresses or CIDR block to which the settings apply. For more information, see Continuous profiling for Java applications.

Information desensitization settings

In the Information Desensitization Settings section, you can set a desensitization rule to desensitize content such as JVM system parameters, K8s YAML, method input parameters, Arthas environment variables, and system variables during data collection. Elements in the rule are separated by commas, and each element represents a case-insensitive regular expression. For example, password is equivalent to the regular expression .*password.*.

When the agent collects data, it filters data keys based on these expressions. If a key matches an expression, the corresponding value is considered sensitive data and is desensitized. As shown in the following figure, if the desensitization rule is set to licenseKey, the value of the key -Darms.licenseKey is desensitized.

The following figure shows the effect after IP desensitization is enabled:

Interface call settings

In the Interface Call Settings section, you can specify a threshold for slow calls and configure the exception whitelist, HTTP status code whitelist, and invalid interface call filtering policy.

• Slow Call Threshold: The default value is 500. If the response time of an interface is greater than this threshold, the interface is marked as a slow call.

• HTTP status code whitelist:

  By default, an HTTP status code greater than or equal to 400 is classified as an error call. If you do not want a specific status code to be classified as an error, you can add it to the whitelist to ignore this type of error.

  This affects only the HTTP frameworks that are supported by Application Monitoring.

  Affected data: Error count metrics for the HTTP server-side or client (arms_http_requests_error_count, arms_http_client_requests_error_count, and arms_app_requests_error_count) and span status.

  Affected features: The error count on the Overview, Provided Services, and Dependencies tabs, and the span status and error count alerts on the Trace Explorer page.

  Content format: Enter a single status code. Separate multiple status codes with commas (,). Fuzzy match is not supported.

  Example: 403,502

  Default value: Empty

• Invalid interface call filtering:

  If you do not want to view specific calls on the Provided Services tab, you can enter the names of the interfaces to filter. The agent will not report observability data for these interfaces, which hides them from the interface call page.

  Note

  • For Java agents earlier than V4.2.0, this feature is effective only for interfaces that appear on the Provided Services tab.

  • For Java agents of V4.2.0 or later, this feature is effective for any LocalRootSpan.

  Affected data: All metrics and spans that correspond to the interface are ignored.

  Affected features: All metrics that correspond to the interface on the Overview, Provided Services, and Dependencies tabs, and the number of spans, number of calls to the interface, number of errors, and slow call alerts on the Trace Explorer page.

  Content format: Use a string or an AntPath expression to match the full name of an invalid interface. Separate multiple rules with commas (,). The AntPath expression in the default value is for compatibility with historical data. We recommend that you do not delete it. Append new configurations after the existing rules.

  Example: /api/test/*,/api/playground/create

  Default value: /**/*.jpg,/**/*.png,/**/*.js,/**/*.jpeg,/**/*.pdf,/**/*.xlsx,/**/*.txt,/**/*.docs,/**/*.gif,/**/*.csv

• Record upstream interface name in interface call metric and Record upstream application name in interface call metric:

  Configuration purpose: Controls whether to record the upstream application and upstream interface that call this interface in the interface metric. This primarily affects whether data exists for upstream and downstream traces on the Provided Services tab. When an application has many upstream applications, recording this information can significantly increase the volume of reported metrics and lead to higher costs.

  Configuration effect: As shown in the figure, after the recording of upstream applications and interfaces in the interface metrics is disabled, the corresponding data drops to 0.

  

• Record original status code in interface call metric:

  Configuration purpose: Records the original response code in HTTP interface-related metrics.

  Configuration effect:

  

• Record HTTP method name for interface: When this switch is enabled, the interface name displayed on the Interface Calls page includes the HTTP request method name. For example, the name changes from /api/v1/use/{userId} to GET /api/v1/use/{userId}.

Database call settings

In the Database Call Settings section, you can specify a threshold for slow SQL statements and a maximum length for collected SQL statements. You can also specify whether to display the variable binding values and constant values in SQL statements, and whether to record the size of the return value for MySQL queries.

• Display variable binding values in SQL: Captures the variable values bound to the PreparedStatement parameter. This configuration takes effect without restarting the application.

  The variable values bound to the PreparedStatement parameter are recorded and stored as attributes in the corresponding database access span. In these attributes, the key is db.bindvalue and the value is a comma-separated list of variable values. This configuration takes effect without restarting the application.

  • This setting is supported by all JDBC-based frameworks that are currently supported by ARMS.

  • The following database variable types are supported for this setting: Boolean, Byte, Short, Int, Long, Float, Double, BigDecimal, and Date.

  • Enabling this switch incurs additional overhead, which is directly proportional to the number of variables in the SQL statement. In official stress testing scenarios (with one variable), enabling this feature results in less than 0.5% additional CPU overhead and negligible memory overhead.

  The following figure shows the actual effect after this switch is enabled:

  

• Display constant values in SQL: This setting displays the constant values in SQL statements without additional processing. This configuration takes effect without an application restart. If you enable this option, SQL statements with different constant values are treated as distinct statements. This can prevent them from being correctly converged and may cause them to be grouped under {ARMS_OTHERS}. We recommend that you keep this option disabled.

Trace context propagation protocol settings

In the Trace context propagation protocol settings section, you can select a trace context propagation protocol based on your requirements.

By default, the ARMS agent selects a trace context propagation protocol based on whether the request header contains specified protocol headers. For more information, see ARMS trace context propagation protocol selection policy.

You can select any protocol on this page as the preferred protocol. After you select and save your preference, ARMS prioritizes checking for the request headers of the selected protocol. For example, with the following configuration, when a request is received, the ARMS agent checks for the protocol context in the order of Jaeger, EagleEye, W3C, SkyWalking, and Zipkin.

You can also choose to enforce the use of a specific protocol. For example, with the following configuration, when a request is received, the ARMS agent checks only for the presence of request headers defined by the Jaeger protocol. If these headers are not detected, it does not check for other protocols. Instead, it generates a new trace context.

Configuration purpose:

This feature supports selecting a preferred or enforced trace context propagation protocol. Common scenarios include the following:

• The request carries trace contexts in multiple formats, and the default parsing order does not meet your requirements. You want to specify a protocol to be prioritized. For example, if a request carries both W3C and Zipkin trace contexts, the W3C trace context is parsed with priority by default. You can set the propagation protocol to Zipkin to prioritize the Zipkin protocol.

• The request carries a trace protocol header, but you do not want to reuse it. You want to generate a new trace context using another protocol. For example, if a request carries a Zipkin trace context, the Zipkin trace context is parsed by default. By setting the propagation protocol to W3C and changing the propagation mode to Enforced, the system ignores the Zipkin trace context and generates a new W3C-compliant context. Note that if you do not change the propagation mode to Enforced, the system still attempts to parse the Zipkin trace context after it fails to parse the prioritized W3C context.

Configuration effect:

After configuration, each LocalRootSpan in a trace will include an attribute with the key trace.protocol.type and a value that indicates the trace protocol of the current trace.

Message queue settings

In the Message Queue Settings section, you can customize consumer information.

• Custom RabbitMQ consumer: You can specify a class name for a custom consumer or a class name for an anonymous internal consumer to view the traces of the consumer. Separate multiple consumers with commas (,).

• Custom Kafka consumption method: You can specify a custom consumption method to view the traces and metrics when you consume messages using the native Kafka SDK.

• Automatic context pass-through for Kafka messages: When a message is sent, multiple headers are automatically added to the Kafka message to associate the sending trace with the consumption trace.

Agent collection settings

In the Agent Collection Settings section, you can set the maximum number of traces collected per second by the agent, the maximum QPS threshold that the agent can handle, and the log level of the agent.

• Collect traces: Controls whether trace data is reported. This feature is enabled by default. If you disable it, trace data is no longer reported.

• Maximum number of traces collected by the agent per second: The number of spans that the agent can report per second. For performance reasons, the actual effective threshold may have a deviation of up to 5% from the user-configured threshold. Spans that exceed this limit are not reported.

• Maximum QPS threshold that the agent can handle: The number of requests that the agent can handle per second. For performance reasons, the actual effective threshold may have a deviation of up to 5% from the user-configured threshold. Requests that exceed this threshold are not monitored. This means that no spans or metrics are generated for these requests, and the feature of associating logs with TraceId does not take effect.

• Collect internal call data without an entry: In ARMS, HTTP services and RPC services provided by applications, scheduled tasks triggered by applications, and message consumption are considered entries. Database calls and HTTP requests sent within the business logic of these entries are considered internal calls with an entry. Conversely, scheduled HTTP calls, database calls, NoSQL calls, message sending, and RPC calls that are initiated using a Java Development Kit (JDK) thread pool are considered internal calls without an entry. Agents of V4.2.2 and later provide a one-click option to filter out this data. As shown in the following figure, in the Lettuce framework, commands are periodically executed to ensure a normal connection with the Redis server-side. Such calls are internal calls without an entry. After the collection of internal call data without an entry is disabled at the time point shown in the figure, the related data is no longer collected.

  

• Agent log level: Adjusts the logging level of the agent, which is useful for troubleshooting.

• Report agent logs: Specifies whether to allow the agent to report its operational logs to the ARMS server-side for quick troubleshooting. Under normal circumstances, only a small amount of logs are reported per minute.

• Default metric collection filter: Filters the reporting of specified types of metrics. You can select multiple types from the following supported types:

  • custom_entry: custom metrics for method monitoring

  • http: metrics for HTTP server-side

  • http_client: metrics for HTTP client

  • rpc: metrics for RPC call server-side

  • rpc_client: metrics for RPC call client

  • sql: metrics related to database service access

  • nosql: metrics related to NoSQL service access

  • cache: metrics related to cache service access

  • producer: metrics related to message sending

  • consumer: metrics related to message consumption

  • schedule: metrics related to scheduling service

• Custom metric collection configuration: Configures the metrics to be collected that are defined by the OpenTelemetry SDK. For more information, see Customize metrics using the OpenTelemetry Java SDK.

• Baggage collection list: When your business uses baggage, this setting writes specific baggage to the span attributes, making it easier to retrieve specific traces.

Advanced exception filtering settings

In the Advanced exception filtering settings section, you can configure exception collection rules.

• Collect plug-in exceptions: Specifies whether to collect plug-in exceptions.

• Stack depth for distinguishing similar exceptions: The default value is 2. This stack depth is used to identify similar types of exceptions. Modifying this configuration may cause unexpected statistical behavior. Proceed with caution.

• Exception filtering whitelist:

  If you do not want to view these exceptions on the Exception Analysis tab, you can enter the fully qualified class names of the exceptions that you want to ignore. This hides them from the Exception Analysis page.

  Affected data: Exception count metrics (arms_exception_requests_count_raw and arms_exception_requests_seconds_raw) and exception information in spans.

  Affected features: Metrics for the corresponding exceptions on the Exception Analysis tab.

  Content format: Use the fully qualified class names to identify the exceptions that need to be filtered. Separate multiple exceptions with commas (,).

  Example: java.lang.InterruptedException,java.lang.IndexOutOfBoundsException

  Default value: None

• Exception filtering for parent class inheritance: If you enable this switch, an exception is also filtered if it is a child class of an exception class that is configured in the exception filtering whitelist.

  Configuration effect: Exceptions that meet the filtering criteria are not displayed in the ARMS console.

• Exception message filtering: After this feature is configured, exceptions of a specified type are also filtered if their message fields match the configured conditions.

  • Exception name: Specifies the exception to which the filter applies.

  • Message condition: The condition for the exception message. The value can be startsWith, endsWith, or contains.

  • Message keyword: The keyword string.

  Configuration effect: Exceptions that meet the filtering criteria are not displayed in the ARMS console.

Pooling monitoring settings

In the Pooling monitoring settings section, you can configure collection rules for thread pools and connection pools.

• Thread pool and connection pool monitoring: Specify whether to monitor thread pool metrics in frameworks such as Tomcat, Dubbo, and High-speed Service Framework (HSF). To enable this feature, you must upgrade the agent to the latest version.

• Thread Pool Thread Name Pattern Extraction Policy: This feature, by default, replaces all numeric characters in the thread names of any running threads within a thread pool with *. You can also adjust this to only replace the ending characters of the thread names with *. In applications where multiple Dubbo Providers are initiated and these providers have different listening ports, if the default strategy is applied, two thread pools from two Dubbo Providers might be aggregated into a single thread pool because their extracted thread name templates are identical. In this case, adjusting the strategy helps distinguish between the two thread pools.

  Configuration effect: For a common Tomcat thread pool, the displayed thread pool thread name is http-nio-*-exec-* by default. After adjustment, the thread name becomes http-nio-9099-exec-*.

  

• Thread pool usage scenario filtering and Thread pool thread name pattern filtering: Metrics for specific thread pools can be excluded from reporting based on their usage scenarios and thread name patterns.

  Note

  This configuration is effective only for Java agents of V4.2.0 or later.

  • Thread pool usage scenario: The scenario in which the thread is used. Currently, Tomcat, Vert.x, Undertow, Dubbo, Jetty, AliyunJavaAgent, and default are supported. AliyunJavaAgent represents the thread pool used by the agent, and default represents other unclassified thread pools.

  • Thread pool thread name pattern: The pattern obtained after the thread names in the thread pool are processed. For example, http-nio-*-exec-* is generally obtained by replacing the numeric part of the actual thread name with *.

  Affected data: Thread pool metrics

  Affected features: The Thread Pool Monitoring tab and alerts configured based on thread pool monitoring metrics.

  Content format:

  • Thread pool usage scenario filtering: The usage scenarios displayed on the Thread Pool Monitoring tab. Separate multiple usage scenarios with commas (,).

  • Thread pool thread name pattern filtering: The thread name patterns displayed on the Thread Pool Monitoring tab. Separate multiple usage scenarios with commas (,). The matching method is exact match. Rule-based matching is not supported.

  Example:

  • Thread pool usage scenario filtering: AliyunJavaAgent,Jetty

  • Thread pool thread name pattern filtering: Catalina-utility-*,DubboServerHandler-*-thread-*

  This indicates that all thread pool data for the AliyunJavaAgent scenario is not reported. For other scenarios, if the thread pool pattern name is Catalina-utility-*, the data is also not reported.

  Default value: None

Span attributes settings

• Record OTel Spec conventional attributes: The OpenTelemetry Specification stipulates the attributes that should be included in the spans generated by each plug-in type. However, to reduce the data reporting volume, the ARMS agent does not record these attributes in spans by default. You can enable this feature based on your needs. For information about the additional attributes that are added by various frameworks after this feature is enabled, see OpenTelemetry Specification.

  The following example shows a span for an HTTP Server. The attributes within the red box are the attributes that are added after this switch is enabled.

  

• Span association application label configuration: This setting controls which spans have application labels attached. Application labels are configured on the Applications page. By default, all spans include application labels. To reduce data volume, you can attach application labels only to entry spans. Entry spans typically include HTTP Server, RPC Server, MQ message consumption, and scheduled tasks.

  The following figure shows the effect after you configure the labels to be attached to entry spans. For example, an application contains a pair of application labels such as test1:value1.

  

  For an entry span, you can see the attributes of the corresponding label.

  

  For a non-entry span, it does not contain the attributes of test1:value.

  

Advanced settings

In the Advanced Settings section, you can configure advanced settings, such as the interfaces that you want to filter and the maximum length of method stacks.

• Quantile Statistics: You can enable or disable quantile statistics.

• Maximum retention length of method stack: The default value is 128 and the maximum value is 400.

• Trace compression: Specifies whether to simplify repeated calls, such as for loops.

• Maximum display length of request input parameters: The default length is 512 characters and the maximum length is 2,048 characters.

• Asynchronous pass-through scan package name: To monitor asynchronous tasks, you can add a scan package for asynchronous propagation to your application configuration. When a `Runnable`, `Callable`, or `Supplier` object is created, a method in the specified scan package automatically captures the trace context of the current thread. This captured context is then propagated to the new thread when the asynchronous task is executed.

• Include TraceId in HTTP response: Specifies whether to return the eagleeye-traceid field in the response headers for HTTP requests.

• JVM GC Trigger Root Cause: Specifies whether to collect the root causes of Java Virtual Machine (JVM) garbage collection (GC) triggers. You can view the monitoring details on the JVM Monitoring page. For information about the GC trigger root causes, see GC-related metrics.

Copy settings to other applications

To synchronize the same settings for other applications, you can copy the corresponding settings to other applications.

Global default configurations

You can save the current settings as global default configurations. When you create a new application, these settings are used by default.

1. At the bottom of the page, click Save current app settings as global default configuration.

2. If the Current settings not saved dialog box appears, click OK to save the settings for the current application, and then click Save current app settings as global default configuration.

3. In the dialog box that appears, click Confirm.

FAQ