Application Real-Time Monitoring Service:Log analysis

Why does log correlation fail after XML changes?

If this issue occurs when you query logs in SLS, first verify that the logs are reported to SLS and that you have bound the correct project and Logstore in the ARMS console.

After you confirm that data exists in SLS, troubleshoot the issue based on the following two scenarios:

• None of the logs are correlated with a trace ID. This issue has several possible causes:

  • The agent is not successfully attached.

    You can verify this by checking whether all metrics are displayed correctly in the ARMS console.

  • The main switch of the agent is disabled, or certain plug-ins, such as the Tomcat plug-in, are turned off.

    You can check this in the Agent switch settings section on the Custom configurations page.

  • The server-side framework is not supported. For example, you might be using an unsupported web container, RPC framework, scheduled task framework, or messaging framework. For a list of supported frameworks, see Java components and frameworks supported by application monitoring.

  • You are using logging frameworks other than Log4j, Log4j2, or Logback, or you have heavily modified these frameworks.

  • The log XML configuration is incorrect.

    Use the following steps to investigate:

    • If you are using agent version 4.1.6 or later, enable the automatic trace ID injection feature. If trace IDs can be correlated, this confirms an error in the log XML configuration.

    • Import the ARMS SDK for Java and manually retrieve the trace ID when you print logs. The following is sample code:

      Span span = Tracer.builder().getSpan();   // This does not create a new span.
      String traceId = span.getTraceId();
      logger.warn("traceId={} this is your log message", traceId)

      If you can retrieve the trace ID, this confirms an error in the log XML configuration.

    • If both methods fail, submit a ticket.

• Some logs are not correlated with a trace ID. This may be because the logs lack a trace context.

  Typically, logs do not have a trace context when they are written. A trace context is usually present only in logs that are printed at request entry points, such as in the processing code for HTTP interfaces, RPC interfaces, scheduled tasks, or message consumption.

  Sample scenarios:

  • After an application receives an HTTP request, it executes a database query and then prints a log. In this case, a trace context exists, and the log can be successfully correlated with a trace ID.

  • An application receives an HTTP request, submits an asynchronous task to a thread pool to execute a database query, and then prints a log.

    • For agent v3.x, logs cannot be correlated with a trace ID because the agent does not support automatic asynchronous propagation of the trace context. The asynchronous thread lacks a trace context.

    • For agent v4.x, logs can be correlated with a trace ID because the agent supports automatic asynchronous propagation of the trace context. The asynchronous thread has a trace context.

  • After startup, an application uses a JDK thread pool to cyclically execute database queries and print a log after each query. In this case, no trace context exists, and the logs cannot be correlated with a trace ID.

  You can identify the specific scenario by checking the name of the thread that wrote the log.

  For example:

  • When you use Tomcat as the web container, thread names that start with http-nio- typically represent threads that handle HTTP requests. Logs printed by these threads can be correlated with a trace ID.

  • When you use Dubbo as the RPC framework, thread names that start with DubboServerHandler- typically represent threads that handle Dubbo requests. Logs printed by these threads can be correlated with a trace ID.

  Other frameworks behave similarly. If you determine that the issue is not caused by any of these scenarios, submit a ticket.

Why are logs empty in Trace Explorer?

Follow these troubleshooting steps:

1. Check if none of the application's logs are correlated with a trace ID. If so, refer to the troubleshooting steps in Why are logs not correlated with trace IDs after I modify the log XML configuration?. If not, proceed to the next step.

2. If you cannot find logs for a trace ID for one or more specific interfaces, this generally means the interfaces did not output any logs.

3. If none of the above scenarios apply, submit a ticket.

Why is the Logstore empty after binding?

You must configure log collection for the Logstore according to the SLS documentation. ARMS does not automatically collect logs into the Logstore.

Why do gateway logs fail trace ID correlation?

The v3.x agent has a known defect in its instrumentation of Spring Cloud Gateway. Upgrading to the v4.x agent resolves this issue.

Why does trace ID correlation fail with TraceCallable.wrap()?

This indicates the method is being used incorrectly. We recommend upgrading to the v4.x agent, which handles asynchronous scenarios without code modification.

Why does trace ID correlation fail with auto-injection?

This issue typically occurs in two situations:

• None of the logs are correlated with a trace ID: This usually indicates that the configuration was not successfully applied to the agent. Check the connectivity to the Application Configuration Management (ACM) server by referring to the Network configuration for Java application monitoring documentation.

• Some of the logs are not correlated with a trace ID: Refer to the troubleshooting steps in Why are logs not correlated with trace IDs after I modify the log XML configuration?.

Can ARMS replace Simple Log Service (SLS)?

No. Application Real-Time Monitoring Service (ARMS) and Simple Log Service (SLS) serve different purposes and complement each other rather than replace each other.

• ARMS is an Application Performance Monitoring (APM) product that focuses on trace analysis, API monitoring, and application performance analysis. The Log analysis feature of ARMS queries logs from the SLS project and Logstore that you bind, so it uses SLS as its log storage backend. ARMS does not collect logs into the Logstore for you.

• SLS is a general-purpose service for log collection, storage, and analysis, and it supports data ingestion from multiple data sources.

To use the Log analysis feature of ARMS, bind an SLS project and Logstore as described in Step 1: Correlate business logs in this topic. Then, in the top navigation bar, choose Scenario-based Analysis > Log Analysis to query and analyze the logs.