Application Real-Time Monitoring Service:Java application custom configuration

Prerequisites

You have installed a probe for your application. For instructions, see Overview of Application Monitoring Integration.

Procedure

1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring > Application List.

2. On the Applications page, select a region at the top of the page and click the name of your target application.

   Note

   The icons in the Language column indicate the following:

   : A Java application integrated with application monitoring.

   : A Go application integrated with application monitoring.

   : A Python application integrated with application monitoring.

   -: An application integrated with Managed Service for OpenTelemetry.

3. In the top navigation bar, choose Application Configuration > Custom Configurations.

4. Set the custom configuration parameters and click Save.

Sampling settings

In the Sampling Settings section, you can configure sampling policies, the sampling rate, and characteristics of always-sampled interfaces for traces. You can also enable or disable low-traffic sampling. For more information, see Select a trace sampling mode (agent V3.2.8 or later).

Agent switch settings

In the Agent Switch Settings section, you can enable or disable application monitoring and toggle individual agent switches.

This section lists all available agent switches in a grid. You can enable or disable monitoring for specific frameworks and features as needed.

Thread profiling settings

In the Thread profiling settings section, you can turn the main switch for thread profiling on or off.

Application log association configuration

In the Application log association configuration section, you can configure log source information to automatically inject TraceId and SpanId. For more information, see Log analysis.

• Automatic TraceId and SpanId injection: When enabled, the agent automatically injects the TraceId and SpanId into your logs without changing your log configuration.

  Effect:

  # log4j
  2024.04.12 17:14:18 WARN  log4j-11 c.e.d2.TraceIdLogWorker {traceId=8b51c17d03cfd6db4a68d8e8da04fc85, spanId=ea4bfd29b6d36039} [TraceId] This is a simple trace log
  17:14:18 WARN [log4j-ThreadPool-11] c.e.d2.TraceIdLogWorker {traceId=8b51c17d03cfd6db4a68d8e8da04fc85, spanId=ea4bfd29b6d36039} [TraceId] This is a log with exception
  java.lang.RuntimeException: null
  	at com.example.demo2.TraceIdLogWorker.doWork(TraceIdLogWorker.java:49)
  	at org.apache.arms.java.util.concurrent.ExecutorServiceInterceptor.call(ExecutorServiceInterceptor.java:71)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  
  # log4j2
  17:14 [Thread.ResolverScan-1] c.e.d2.TraceIdLogWorker {traceId=8b51c17d03cfd6db4a68d8e8da04fc85, spanId=ea4bfd29b6d36039} This is a log with exception
  java.lang.RuntimeException: null
  	at com.example.demo2.TraceIdLogWorker.doWork(TraceIdLogWorker.java:49)
  	at org.apache.arms.arthas.resolver.DefaultArtifactResolverChain$DefaultTask.execute(DefaultArtifactResolverChain.java:115)
  
  # logback
  17:14:18 WARN [logback-ThreadPool-1] c.e.d2.TraceIdLogWorker {traceId=8b51c17d03cfd6db4a68d8e8da04fc85, spanId=ea4bfd29b6d36039} [TraceId] This is a simple trace log
  17:14:18 WARN [logback-ThreadPool-1] c.e.d2.TraceIdLogWorker {traceId=8b51c17d03cfd6db4a68d8e8da04fc85, spanId=ea4bfd29b6d36039} [TraceId] This is a log with exception
  java.lang.RuntimeException: null
  	at com.example.demo2.TraceIdLogWorker.doWork(TraceIdLogWorker.java:49)

• Add TraceId to log MDC: By default, agent V4.x and later do not support obtaining the TraceId by using org.slf4j.MDC.get("EagleEye-TraceID"). To use this method, enable this switch.

URL convergence settings

In the URL Convergence Settings section, you can enable or disable URL convergence and configure the convergence threshold and rules. URL convergence groups a series of similar URLs and displays them as a single entry. For example, it groups URLs that start with /service/demo/. The convergence threshold is the minimum number of URLs required to trigger convergence. For example, if the threshold is 100, a group of URLs matching the rule's regular expression is converged only after the count reaches 100. For more information, see How ARMS convergence works.

Arthas monitoring

In the Arthas Monitoring section, you can enable or disable the Arthas diagnostics feature and specify the IP addresses where it should be active. For more information, see Arthas diagnostics.

Continuous profiling settings

In the Continuous profiling settings section, you can turn the main switch, CPU hotspot, memory hotspot, and code hotspot features on or off. You can also specify the IP addresses or CIDR blocks where profiling should be active. For more information, see Continuous profiling for Java applications.

Data desensitization settings

In the Data Masking Settings section, you can configure desensitization rules. The agent then desensitizes data such as JVM system parameters, Kubernetes YAML files, method input parameters, and Arthas environment and system variables during collection. In the desensitization rules, elements are separated by commas. Each element represents a case-insensitive regular expression. For example, password is equivalent to the regular expression .*password.*.

When the agent collects data, it filters data keys against these expressions. If a key matches a rule, the agent treats it as sensitive and masks its value. For example, if you set the rule to licenseKey, information with the key -Darms.licenseKey is desensitized.

After you enable IP desensitization, the agent automatically desensitizes information involving IP addresses, such as endpoint addresses on various pages in the application monitoring console. This process does not affect how monitoring data is displayed.

API call configuration

In the Interface call configuration section, you can configure policies such as the slow call threshold, exception whitelists, HTTP status code whitelist, and invalid interface call filtering.

• Slow call threshold: The default value is 500 ms. An interface call is marked as slow if its response time exceeds this threshold.

• HTTP status code whitelist:

  By default, HTTP status codes of 400 or greater are classified as error calls. If you do not want to classify certain status codes as errors, you can add them to the whitelist.

  This setting only affects the HTTP frameworks currently supported by application monitoring.

  Affected data: Error count metrics for HTTP servers and clients (arms_http_requests_error_count, arms_http_client_requests_error_count, and arms_app_requests_error_count) and span status.

  Affected features: The error counts on the Overview, Provided Services, and Dependencies tabs, as well as the span status and error count alerts on the Trace Explorer page.

  Format: Enter a single status code. To enter multiple codes, separate them with commas (,). Fuzzy matching is not supported.

  Example: 403,502

  Default: Empty

• Invalid interface call filtering:

  If you want to hide certain calls from the Provided Services tab, you can enter the names of the interfaces you want to ignore. The agent will not report observability data for these interfaces, and these interfaces will not appear on the interface call page.

  Note

  • For Java agents earlier than V4.2.0, this feature is effective only for interfaces that appear on the Provided Services tab.

  • For Java agent V4.2.0 and later, this feature is effective for any LocalRootSpan.

  Affected data: All metrics and spans for the matching interfaces are ignored.

  Affected features: All metrics for the matching interfaces on the Overview, Provided Services, and Dependencies tabs. The number of spans, call volume, error count, and slow call alerts for the matching interfaces on the Trace Explorer page are also affected.

  Format: Use a string or an AntPath expression to match the full name of an invalid interface. To enter multiple rules, separate them with commas (,). The default value is written as an AntPath expression to ensure compatibility with existing data. We recommend that you do not delete it. Append new configurations to the existing rules.

  Example: /api/test/*,/api/playground/create

  Default: /**/*.jpg,/**/*.png,/**/*.js,/**/*.jpeg,/**/*.pdf,/**/*.xlsx,/**/*.txt,/**/*.docs,/**/*.gif,/**/*.csv

• Record upstream interface name in interface call metrics and Record upstream application name in interface call metrics:

  Purpose: Controls whether interface metrics record the upstream application and interface that made the call. This primarily affects whether data for upstream and downstream services is available. If an application has many upstream applications, recording this information can significantly increase the volume of reported metrics and lead to higher costs.

  Effect: After you disable the recording of upstream applications and interfaces for interface metrics, the corresponding data count drops to zero.

• Record original status code in interface call metrics:

  Purpose: Records the original response code in HTTP interface-related metrics.

  Effect:

  After this is enabled, you can view the original response code of HTTP interfaces in the interface call details. This enables more detailed statistics and analysis of status codes.

• Record HTTP method name for interface: After you enable this switch, the expanded interface name on the interface call page includes the HTTP request method. For example, the name changes from /api/v1/use/{userId} to GET /api/v1/use/{userId}.

Database call configuration

In the Database call configuration section, you can configure the slow SQL threshold, the maximum length of collected SQL statements, and specify whether to display variable-bound and constant values in SQL statements, and whether to record the size of MySQL query return values.

• Display bound variable values in SQL: Captures variable values bound to PreparedStatement parameters. This change takes effect without restarting the application.

  The agent records PreparedStatement parameter values in the attributes of the corresponding database call span. The key is db.bindvalue, and the value is a comma-separated list of variable values. This change takes effect without restarting the application.

  • This setting is supported by all JDBC-based frameworks that are currently supported by ARMS.

  • Supported database variable types: Boolean, Byte, Short, Int, Long, Float, Double, BigDecimal, and Date.

  • Enabling this switch incurs additional overhead proportional to the number of variables in the SQL statement. In official stress test scenarios with one variable, enabling this switch adds less than 0.5% CPU overhead and negligible memory overhead.

  The following figure shows the effect after this switch is enabled:

  After this is enabled, you can view the db.bindvalue attribute and its corresponding variable values in the Span Details panel of a database call span.

• Display constant values in SQL: The agent only truncates SQL statements and does not apply extra processing. This change takes effect without restarting the application. If SQL statements vary only in constant values, they may become too fragmented and end up converged into {ARMS_OTHERS}. We recommend that you do not enable this option.

• Record original JDBC connection string: By default, the agent can parse only a limited number of JDBC connection string formats. For connection strings that the agent cannot parse correctly, it records the endpoint as UnknownEndpoint. If you enable this switch, the agent records the original JDBC connection string. For example, a jdbc:h2:file connection string that was previously unrecognized and recorded as UnknownEndpoint is correctly recorded after this switch is enabled.

Trace context propagation settings

In the Trace Context Propagation Protocol Settings section, you can select the trace context propagation protocol that fits your needs.

By default, the ARMS agent selects a trace context propagation protocol based on whether the headers of a request contain specific protocol headers. For more information, see ARMS trace context propagation protocol selection policy.

On this page, you can select any protocol as the priority protocol. After you select and save the configuration, ARMS prioritizes detecting the request headers defined by that protocol. For example, with the following configuration, the agent detects protocol contexts in the order of Jaeger, EagleEye, W3C, SkyWalking, and Zipkin for incoming calls.

You can also choose to enforce a specific protocol. For example, with the following configuration, the agent only detects request headers defined by the Jaeger protocol for incoming calls. If the headers are not found, it does not check for other protocols but generates a new trace context instead.

Trace context extraction and injection strategy: The default behavior is to extract the trace context from upstream services and inject it into downstream requests. You can configure the extraction and injection behavior with this setting.

Purpose:

You can select a priority or enforced trace context propagation protocol. Common use cases include:

• A request carries trace contexts in multiple formats, and the default parsing order does not meet your requirements. You want to prioritize a specific protocol. For example, if a request carries both W3C and Zipkin trace contexts, the agent parses the W3C trace context first by default. You can set the propagation protocol to Zipkin to parse the Zipkin protocol with priority.

• A request carries a trace protocol header, but you do not want to reuse it and want to generate a new trace context based on another protocol. For example, if a request carries a Zipkin trace context, the Zipkin trace context is parsed by default. You can set the propagation protocol to W3C and the propagation mode to Enforced. This way, the Zipkin trace context is ignored, and a new context is generated based on the W3C protocol specification. Note that if the propagation mode is not set to Enforced, the agent still attempts to parse the Zipkin context after the prioritized W3C context parsing fails.

Effect:

After configuration, each LocalRootSpan has a trace.protocol.type attribute that indicates the trace protocol used.

Message queue configuration

In the Message queue configuration section, you can customize consumer information.

• Custom RabbitMQ consumer: Specify the class name of a custom consumer or a class that contains an anonymous inner consumer to view the trace of that consumer. Separate multiple consumer class names with commas (,).

• Custom Kafka consumption method: Define a custom consumption method to view traces and metrics when consuming messages with the native Kafka SDK.

• Automatically propagate context for Kafka messages: When sending a message, automatically add headers to the Kafka message to correlate the producer and consumer traces.

Agent collection configuration

In the Agent collection configuration section, you can set the maximum number of spans collected per second, the maximum QPS threshold that the agent can handle, and the agent log level.

• Collect traces: Controls whether trace data is reported. This feature is enabled by default. If you disable it, the agent stops reporting trace data.

• Maximum spans collected per second: The number of spans that the agent can report per second. For performance reasons, the actual effective threshold may deviate from the configured threshold by up to 5%. Spans that exceed the limit are not reported.

• Maximum QPS threshold for the agent: The number of requests that the agent can handle per second. For performance reasons, the actual effective threshold may deviate from the configured threshold by up to 5%. The agent does not monitor requests that exceed this threshold. Consequently, no spans are generated and no metrics are recorded for these requests, and the log-TraceId association feature does not take effect.

• Collect internal call data without an entry point: In ARMS, an entry point can be an HTTP service, an RPC service, a scheduled task, or message consumption. Calls such as database calls and outbound HTTP requests that occur within the business logic of these entry points are considered internal calls with an entry point. By contrast, periodic HTTP calls, database calls, NoSQL calls, message sending, and RPC calls initiated by JDK thread pools are considered internal calls without an entry point. In agent V4.2.2 and later, you can filter this type of data with one click. For example, the Lettuce framework periodically runs commands to keep the connection to Redis healthy. These are internal calls without an entry point. After you disable collection, the agent no longer collects the related data.

  For example, on the NoSQL Analysis page, call data generated by Redis commands such as PING, QUIT, and AUTH that are automatically executed by the Lettuce framework will no longer be displayed.

• Agent log level: Adjusts the logging level of the agent for troubleshooting.

• Report agent logs: Allows the agent to report its operational logs to the ARMS server for quick troubleshooting. Under normal circumstances, the agent reports only a small number of logs per minute.

• Enable HTTPS reporting: When enabled, the agent uses HTTPS to report data.

• Default metric collection filter: Filters the reporting of specified types of metrics. The following types are currently supported, and you can select multiple options:

  • custom_entry: Custom metrics for method monitoring.

  • http: HTTP server metrics.

  • http_client: HTTP client metrics.

  • rpc: RPC server metrics.

  • rpc_client: RPC client metrics.

  • sql: Metrics related to database access.

  • nosql: Metrics related to NoSQL service access.

  • cache: Metrics related to cache service access.

  • producer: Metrics related to message sending.

  • consumer: Metrics related to message consumption.

  • schedule: Metrics related to scheduling services.

• Custom Metric Collection Configuration: Configures the collection of metrics defined by the OpenTelemetry SDK. For more information, see Customize metrics by using the OpenTelemetry Java SDK.

• Baggage to collect: If your application uses baggage, this setting writes specific baggage items into span attributes, making it easier to retrieve specific traces.

Advanced exception filtering configuration

In the Advanced exception filtering configuration section, you can configure exception collection rules.

• Collect plugin exceptions: Specifies whether to collect plugin exceptions.

• Stack depth for differentiating similar exceptions: The default value is 2. This stack depth is used to identify similar types of exceptions. Modifying this setting may cause unexpected statistical behavior. Proceed with caution.

• Exception filtering whitelist:

  If you do not want to see certain exceptions on the Exception Analysis tab, you can enter the fully qualified class names of exceptions that you want to hide. These exceptions are not counted and do not appear on the Exception Analysis page.

  Affected data: Exception count metrics (arms_exception_requests_count_raw and arms_exception_requests_seconds_raw) and exception information in spans.

  Affected features: The metrics for corresponding exceptions on the Exception Analysis tab.

  Format: Use the fully qualified class name to identify an exception to be filtered. To filter multiple exceptions, separate their class names with commas (,).

  Example: java.lang.InterruptedException,java.lang.IndexOutOfBoundsException

  Default: None

• Inherit from parent class for exception filtering: When enabled, if a collected exception is a subclass of an exception class in the Exception filtering whitelist, the agent also filters it.

  Effect: Exceptions that meet the filter criteria are not displayed in the ARMS console.

• Exception message filtering: After this is configured, exceptions of a specified type are also filtered if their message field meets the configured conditions.

  • Exception name: The exception type to which the filter applies.

  • Message condition: The condition for the exception message, which can be startsWith, endsWith, or contains.

  • Message keyword: The keyword string.

  Effect: Exceptions that meet the filter criteria are not displayed in the ARMS console.

Pool monitoring configuration

In the Pool monitoring configuration section, you can configure collection rules for thread pools and connection pools.

• Thread pool and connection pool monitoring: Enables monitoring of thread pool metrics for frameworks such as Tomcat, Dubbo, and HSF. You must upgrade the agent to the latest version.

• Thread pool thread name pattern extraction policy: By default, this option replaces all numeric characters in the name of any running thread in a thread pool with an asterisk (*). You can also configure it to replace only the trailing numeric characters with an asterisk (*). If your application starts multiple Dubbo providers on different ports, the default policy may aggregate their distinct thread pools into one because the extracted thread name patterns are identical. Adjust this policy to distinguish them.

  Effect: For a typical Tomcat thread pool, the displayed thread name pattern is http-nio-*-exec-* by default. After adjustment, it becomes http-nio-9099-exec-*.

• Thread pool usage scenario filtering and Thread pool thread name pattern filtering: Filter out the monitoring metrics of certain thread pools based on their usage scenarios and thread name patterns.

  Note

  This configuration is effective only for Java agent V4.2.0 and later.

  • Thread pool usage scenario: The context in which the thread pool is used. Supported scenarios include Tomcat, Vert.x, Undertow, Dubbo, Jetty, AliyunJavaAgent, and default. AliyunJavaAgent represents the thread pool used by the agent itself, and default represents other unclassified thread pools.

  • Thread pool thread name pattern: The pattern derived from the names of threads in the pool. For example, http-nio--exec- is a pattern typically obtained by replacing the numeric parts of the actual thread names with an asterisk (*).

  Affected data: Thread pool metrics

  Affected features: The Thread Pool Monitoring tab and alerts configured based on thread pool metrics.

  Format:

  • Thread pool usage scenario filtering: Enter the usage scenarios displayed on the thread pool monitoring tab. To enter multiple scenarios, separate them with commas (,).

  • Thread pool thread name pattern filtering: Enter the thread name patterns displayed on the thread pool monitoring tab. To enter multiple patterns, separate them with commas (,). This uses exact matching and does not support rule-based matching.

  Example:

  • Thread pool usage scenario filtering: AliyunJavaAgent,Jetty

  • Thread pool thread name pattern filtering: Catalina-utility-*,DubboServerHandler-*-thread-*

  This means the agent will not report any thread pool data for the AliyunJavaAgent scenario. For other scenarios, if a thread pool's name pattern is Catalina-utility-*, the agent will also not report it.

  Default: None

Span attributes configuration

• Record OTel Specification attributes: The OpenTelemetry Specification defines the attributes that a span for each plugin type should contain. To reduce data reporting volume, the ARMS agent does not record all of these attributes by default. You can enable this option based on your requirements. For more information about the attributes that are added for each framework after this option is enabled, see the OpenTelemetry Specification.

  For an HTTP server span, enabling this switch adds the following attributes.

  The added attributes include HTTP information such as http.path, http.scheme, and http.target, as well as network information such as net.host.name, net.host.port, net.protocol.name, net.protocol.version, net.sock.peer.addr, and net.sock.peer.port.

• Span association with application tags: Controls which spans are tagged with the application tags that you have bound on the Applications page. By default, all spans include application tags. To reduce usage volume, you can choose to attach application tags only to entry spans. Entry spans typically include HTTP Server, RPC Server, message queue consumption, and scheduled tasks.

  For example, if an application has a test1:value1 application tag and you configure it to be attached only to entry spans:

  For an entry span, you can see the attributes for the corresponding tag.

  For a non-entry span, it does not contain the test1:value1 attribute.

Advanced settings

In the Advanced settings section, you can configure settings such as interfaces to filter and the maximum method stack length.

• Quantile statistics: Specifies whether to enable quantile statistics.

• Maximum method stack length: The default value is 128 frames, and the maximum value is 400 frames.

• Trace compression: Specifies whether to simplify repetitive calls, such as those in a for loop.

• Maximum display length of request input parameters: The default length is 512 characters, and the maximum supported length is 2,048 characters.

• Asynchronous context propagation scan package: Add a scan package for asynchronous context propagation to monitor asynchronous tasks. When a new object of a Runnable, Callable, or Supplier interface in the package is created, it automatically captures the trace context of the current thread. This context is then used when the asynchronous thread executes, completing the trace.

• Include TraceId in HTTP response: For HTTP requests only, this returns the eagleeye-traceid field in the response header.

• JVM GC trigger root cause: When enabled, you can view the root cause of JVM garbage collection (GC) triggers. You can view the monitoring details on the JVM Monitoring page. For more information about GC trigger causes, see GC-related metrics.

Copy configurations to other applications

If you need to apply the same configuration to other applications, you can copy the settings.

Global default configuration

You can save the current configuration as the global default configuration. This configuration then applies by default when you create new applications.

1. At the bottom of the page, click Save Current App Settings as Global Default Configuration.

2. If the Current settings not saved dialog box appears, click OK to save the configuration for the current application, and then click Save Current App Settings as Global Default Configuration again.

3. In the dialog box that appears, click Confirm.

FAQ