After you install an Application Real-Time Monitoring Service (ARMS) agent for an application, ARMS begins monitoring the application. On the Provided Services page, you can view information about services provided by the application, such as interface calls, message subscriptions, and scheduled tasks.
Prerequisite
An ARMS agent is installed for the application.
Application Monitoring provides a new application details page for users who have enabled the new billing mode.
If you have not enabled the new billing mode, click Switch to New Version on the Application List page to view the new application details page.
View services provided by an application
Log on to the ARMS console. In the left-side navigation pane, choose .
Select a region in the top navigation bar and click an application.
NoteIcons in the Language column indicate the application's programming language:
: Java
: Go
: Python- (Hyphen): an application monitored in Managed Service for OpenTelemetry
In the top navigation bar, click Provided Services.
In the Quick Filter section (icon 1), you can query charts and services by Request Type, Operation, or Host.
In the trend charts section (icon 2), you can view the time series of the number of requests, number of errors, and average time consumed.
Click the
icon to view metric data for a specific period or compare metric data during the same time period across different dates in the dialog box that appears. You can click the 
icon to switch between column chart and trend chart displays.In the service list section (icon 3), you can view the name, request type, and key metrics of each interface defined by the RED Method: rate, errors, and duration.
In the service list, you can perform the following operations:
Click an interface name or click Details, SQL analysis, or NoSQL analysis in the Actions column to view the details of the service. For more information, see Interface details.
Click Traces in the Actions column to view the trace details of an interface. For more information, see Trace Explorer.
Supported frameworks
Interface details
Interface calls
Overview
On the Overview tab, you can view the number of requests, number of errors, average duration, HTTP status codes, and time series of slow calls.
SQL and NoSQL analysis
On the SQL analysis and NoSQL analysis tabs, you can view the lists of SQL and NoSQL requests initiated by the interface. You can also query databases by host. On these tabs, you can locate slow SQL statements or NoSQL statements that cause slow responses of the service.
You can click a database name of SQL or NoSQL to view the details of the database. You can also click Traces on the right side of SQL or NoSQL to view the complete code trace of the SQL or NoSQL execution logic. For more information, see Trace Explorer.
Upstream and downstream interface calls
Upstream and Downstream tabs list the interfaces of upstream applications (those that call the application) and downstream applications (those that are called by the application) and their call performance metrics, including the number of requests, number of errors, and duration information.
Trace explorer
Trace Explorer allows you to combine filter conditions and aggregation dimensions for real-time analysis based on the stored full trace data. This can meet custom diagnostics requirements in various scenarios. For more information, see Trace Explorer.
Message subscription
Message subscriptions of Python applications cannot be viewed.
Overview
On the Overview tab, you can view the number of requests, number of errors, average duration, and consumption delay (currently only supported for RocketMQ 4.8.0+).
SQL and NoSQL analysis
On the SQL analysis and NoSQL analysis tabs, you can view the lists of SQL and NoSQL requests initiated by the interface. You can also query databases by host. On these tabs, you can locate slow SQL statements or NoSQL statements that cause slow responses of the service.
You can click a database name of SQL or NoSQL to view the details of the database. You can also click Traces on the right side of SQL or NoSQL to view the complete code trace of the SQL or NoSQL execution logic. For more information, see Trace Explorer.
Consumption statistics
The Consumption statistics tab lists the consumption details of topics from the message consumer's perspective, including the number of requests, number of errors, and duration.
Trace explorer
Trace Explorer allows you to combine filter conditions and aggregation dimensions for real-time analysis based on the stored full trace data. This can meet custom diagnostics requirements in various scenarios. For more information, see Trace Explorer.
Scheduled tasks
Only the scheduled tasks of Java applications can be viewed.
Overview
On the Overview tab, you can view the number of requests, number of errors, average duration, and time series of scheduling latency.
SQL and NoSQL analysis
On the SQL analysis and NoSQL analysis tabs, you can view the lists of SQL and NoSQL requests initiated by the interface. You can also query databases by host. On these tabs, you can locate slow SQL statements or NoSQL statements that cause slow responses of the service.
You can click a database name of SQL or NoSQL to view the details of the database. You can also click Traces on the right side of SQL or NoSQL to view the complete code trace of the SQL or NoSQL execution logic. For more information, see Trace Explorer.
Downstream interface calls
Downstream tab lists the interfaces of downstream applications (those that are called by the application) and their call performance metrics, including the number of requests, number of errors, and duration information.
Trace explorer
Trace Explorer allows you to combine filter conditions and aggregation dimensions for real-time analysis based on the stored full trace data. This can meet custom diagnostics requirements in various scenarios. For more information, see Trace Explorer.
FAQ
FAQ about interface calls
What does the Upstream tab of the interface details page show?
The Upstream tab of the interface details page shows details about upstream services that call the current application and are being monitored by an ARMS agent. If the application calls itself, it is also displayed as an upstream service on the Upstream tab.
Why are the upstream services of some interfaces not displayed on the Upstream tab?
If the upstream services are not monitored by an ARMS agent, they are not displayed on the Upstream tab.
Why is the upstream and downstream service data disordered?
By default, if the SkyWalking component is installed in the trace, SkyWalking is regarded as the tracing protocol of the entire trace. The ARMS agent earlier than v4.2.x is not fully compatible with the SkyWalking protocol, which causes upstream and downstream service recognition errors. You can upgrade the agent to v4.2.x or later, or configure a protocol for propagation in the Call Chain Pass-through Protocol Settings section of the Custom Configurations page. For more information, see Trace context propagation protocol settings.
Why are traces of error calls missing?
The ARMS agent earlier than v3.2.x may not correctly record the status codes of Dubbo traces. The ARMS agent 3.2.x and later has fixed the issue.
Why are traces of abnormal calls missing?
The ARMS agent earlier than v4.1.x records only traces of error and slow calls, and does not record those of abnormal calls, whereas the ARMS agent v4.1.x and later records traces of slow, abnormal, and error calls.
Why did interface traffic drop?
Determine if interface traffic has actually dropped. This can be assessed by examining whether concurrent changes happen in metrics like CPU utilization and network I/O before and after the reported drop occurred:
If interface traffic has actually dropped, it was caused by service traffic decrease.
If interface traffic has not dropped, the ARMS server may have encountered errors. We recommend that you submit a ticket.
Why is the traffic data of Spring Cloud Gateway not correct?
The ARMS agent earlier than v4.x has deficiencies in the instrumentation of Spring Cloud Gateway. We recommend that you upgrade the agent to v4.x.
How do I query the slow SQL statements of an interface?
Perform the following operation:
New console
On the monitoring details page, click the name of the target interface, and then view on the SQL Analysis page.
Old console
View on the page of the target application.
What does the yellow dot between two interfaces on the Interface Invocation page of the old console mean?
Move the pointer over the dot to check the prompt.
A yellow dot indicates a slow call. By default, a call that takes more than 500 milliseconds is identified as a slow call. You can change the duration threshold in the Interface call configuration section of the Custom Configuration page.
A red dot indicates an error call.
Why is some 5xx status code data not collected?
Per RFC7231, status codes like 502 and 504 are error codes emitted by the Gateway. These status codes might not reflect the actual response codes of the services it calls. In scenarios where only the services connected to the Gateway are monitored by an ARMS agent and the Gateway itself is not, 5xx responses generated by the Gateway are not tracked or reported.
What is "NetWork And Dubbo Response Decode"?
It is a span that is used to count the time consumed by deserialization at the Dubbo transport layer.
Why do interface names contain characters such as * or ARMS keywords?
As interface names are divergent, ARMS converges them by replacing the different parts of similar interfaces. For more information, see ARMS convergence policies.
Why are the external interface call duration and downstream service duration different?
The server-side processing time refers to the duration taken by the server to handle a request after it has been received by the interface. On the other hand, the client-side processing time is measured from when the request is initiated and includes additional time for connection establishment and network latency. The ARMS console separately displays the server-side processing time for handling requests and the total client-side time from when the request is initiated. However, the network-related latency cannot be precisely determined.
Why is the number of exceptions of a normal interface call is greater than 0?
The exceptions were thrown internally by certain frameworks (such as OkHttp3) and were caught externally by the framework itself. Therefore, the business logic remained unaffected. Because ARMS has instrumented this framework, it also recorded these exception details. This does not indicate an actual issue with the business operations. You can evaluate whether these exceptions require any action or remediation. The exception statistics ignore the Message information of the exception. If you want to view the Message information of a specific exception, you can click the corresponding trace and then click the 
icon on the right side of the interface to view the exception information in the method stack.
Why are some interface names /error, /404, /*, and /**?
This is because ARMS was unable to find appropriate routes for the following unexpected requests:
A non-existent URL was requested.
An error was reported when the request verification failed because the request header was invalid. For example, the header contained invalid characters.
An HTTP server without routing configurations was used, such as Netty.
Why is the number of calls displayed on the Provided Services tab different from that displayed on the Trace Explorer tab?
Data provided by Trace Explorer is calculated based on the sampled trace data and varies with the sampling rate.
Why are the duration quantiles displayed on the Provided Services tab different from those displayed on the Trace Explorer tab?
Quantiles on the Provided Services tab are calculated per machine based on bucketing, whereas quantiles provided by Trace Explorer are calculated based on the duration of spans related to the interface after sampling. For more information, see ARMS metric quantile calculation.
Why did the interface calls increase by about two times?
This is because an ARMS agent v3.x and an ARMS agent v4.x were both installed for the application when some interface calls were called. You can go to the page, select any agent, and click View Details to view JVM parameters.
As shown in the following figure, two -javaagent parameters exist, which indicates that an ARMS agent v3.x and an ARMS agent v4.x are both installed.
Why is the number of calls in the ARMS console is inconsistent with that provided by a third-party observability tool?
This may happen in various scenarios. Examples:
Suppose that the number of interface calls per minute for an application calculated by ARMS is 500, whereas that calculated by NGINX is 200. In general, this is most likely caused by traffic bypassing NGINX to directly access the application.
Suppose that the number of times that an application accesses a database per minute is 400, whereas the number of executions on the database per minute is 1,000. Generally, the database is accessed by multiple applications. The ARMS console can only view the page views of specific applications.
How does ARMS support Spring WebFlux?
The ARMS agent earlier than v4.x only supports native Spring WebFlux and Spring Cloud Gateway. WebFlux-related plug-ins that have been modified by using custom web handlers, such as Apache ShenYu, are not supported.
Why do I see applications that don't belong to me in the upstream and cannot navigate to view their detailed information?
This is normal if your application provides services to external parties. The upstream application belongs to another user, meaning there is a cross-user call relationship. In this case, ARMS will record the application name of the upstream application and the number of calls to this interface, but due to user information isolation restrictions, you cannot view the details of another user's application.
Why can't I see the expected data through interface upstream and downstream when there is a clear upstream and downstream call relationship between applications?
ARMS will first randomly check the user's call traces and focus on the trace.protocol.type field recorded in the LocalRootSpan. If the field value is Jager, upstream and downstream recognition will fail due to case conversion issues. If you encounter this problem, please switch to another protocol or submit a ticket for resolution.
FAQ about message subscription
What is the difference between the data on the Message Subscription/Message Publishing page and the data from the self-monitoring panel of the message queue instance?
The data on the Message Subscription/Message Publishing page reflects various performance metrics when the selected application acts as a consumer or producer, presenting monitoring data from the perspective of the client. The monitoring panel data for the message queue instance, on the other hand, shows performance metrics related to topics, which is data from the perspective of the server.
What are the SpanName formats for sending and receiving messages?
Message type | ARMS agent earlier than v4.x | ARMS agent v4.x and later |
RocketMQ/ONS message sending | No span is created. Only method stacks are recorded. |
|
RocketMQ/ONS message receiving |
|
|
Kafka message sending | No span is created. Only method stacks are recorded. |
|
Kafka message receiving |
|
|
RabbitMQ message sending | No span is created. Only method stacks are recorded. |
|
RabbitMQ message receiving |
|
|
Does the ARMS agent for Java support Spring Cloud Alibaba RocketMQ?
The ARMS agent v4.x and later supports this framework. If the agent version is earlier than 4.x, upgrade it.
What is the meaning and unit of the "Message Delay" metric in message queue monitoring?
Message delay refers to the total time taken from when a message is generated until it is consumed by the Consumer. The unit for this metric is milliseconds. Currently, this metric is only supported for collection by RocketMQ Client and ONS Client.
Why does the application have consumer logic but no associated trace or metric data for the consumers?
It might be because the agent version is lower than 4.x, and lambda expressions were used to define the consumption logic within the messageListener.
Due to class enhancement failures for lambda expressions in agent versions 2.x and 3.x, you can avoid this issue by upgrading the agent to v4.x.
Why did I receive the "Magic v1 does not support record headers" error message when I connected Kafka Clients to the ARMS agent for Java?
The ARMS agent for Java uses the Header field of Kafka messages to pass trace context. However, in Kafka versions earlier than 0.11.0.0 (both Client and Broker), the Kafka message body does not support Header fields. If your Kafka-broker or Kafka-clients are below this version, please migrate to version 0.11.0.0 or higher as soon as possible.
If you cannot migrate at this time, for agents earlier than v4.x, you can disable the enhancement of the Kafka component (the kafka-plugin switch) in the Probe switch settings section on the Custom Configuration tab. For agents v4.x and later, you can add the JVM parameter -Dotel.instrumentation.kafka.producer-propagation.enabled=false when starting the service to prevent such issues.
Why are traces missing Spans for receiving or sending messages?
Check whether the message queue Client SDK used by your Producer and Consumer applications is within the range supported by ARMS. For more information, see Java components and frameworks supported by ARMS.
Make sure that both your Producer and Consumer applications are being monitored by Application Monitoring or Managed Service for OpenTelemetry, with agent switches and plug-in switches correctly enabled, and that they report to the same region. Spans for sending messages are generated in the Producer, while Spans for receiving messages are generated in the Consumer. If an application has not been properly monitored, the corresponding Spans will be missing.
If you cannot see Spans for sending messages, check whether the agent version of the Producer application is 4.x or later. In agent versions earlier than 4.x, sending messages does not generate a separate Span but is instead recorded within the method stack of the parent Span. You can click the
icon to the right of the parent Span to view the actual call method stack. 
If you cannot see Spans for receiving messages, check whether Invalid Interface Filter is enabled in the Interface Call Configurations section of the Custom Configuration tab. This option filters out Spans that you are not concerned with.
Why is there a large difference in Kafka response time between agent versions 4.x and earlier?
Issue: Response time changed from several seconds to 0.x milliseconds.
Cause:
In agent version 3.x and earlier, when processing BatchMessageListener, a batch of messages is considered as one call, and all time consumption is counted together.
In agent version 4.x and later, these operations are separated, with each message counted as one call. Therefore, when upgrading from agent version 3.x to 4.x, it is expected that the time consumption per call drops significantly while the number of calls increases significantly.
The statistical method in version 3.x was unreasonable, and version 4.x has been optimized.
Why does the Kafka consumption process appear duplicated in the trace in message subscription?
If you use the batchMessageListener provided by spring-kafka, the processing of each message in this batch will be recorded in the same Kafka consumption logic. The implementation methods of spring-kafka's batchMessageListener include the following:
Adding the
@KafkaListener(topics = "my-topic", batch = true)annotation to the consumption method.The consumption method class implements the
BatchMessageListenerinterface.Enabling batch consumption mode in the Kafka Message Listener Container properties:
containerProperties.setBatchListener(true).
FAQ about scheduled tasks
Why doesn't invalid interface filtering work for scheduled task interfaces?
Check your application's agent version. For agent versions lower than 3.2.0 or between 4.0.0 and 4.2.0, invalid interface filtering may not work. We recommend that you upgrade the agent to version 3.2.10 or higher than 4.2.0.
Why can't I see external call records in the trace when there are external calls in scheduled tasks?
Troubleshooting:
If the scheduled task call record also doesn't exist, first check Java components and frameworks supported by ARMS Application Monitoring, and check agent switch settings.
If only external call records are missing, and the agent version is lower than 4.0.0, it might be due to trace context propagation failure in asynchronous scenarios. Please upgrade the agent to version 4.0.0 or higher, or use the Advanced Settings feature to configure automatic context propagation.
Why is data from the xxl-job framework missing?
Agent versions lower than 3.1.0 do not support the xxl-job framework. Please upgrade the agent to a higher version.
If you have customized the xxl-job code, such as modifying the fully qualified class names of certain key classes (including class names and package names), the agent instrumentation will not work. In this case, we recommend that you add custom instrumentation to the trace using the OpenTelemetry Java SDK.