Application Real-Time Monitoring Service:Log analysis

Why did trace IDs fail to associate with logs after I modified the XML configurations?

If you encounter this problem when querying logs in SLS, check whether the logs have been reported to SLS, and whether the SLS project and Logstore have been specified for association in the ARMS console.

If data exists in SLS, perform the following operations to troubleshoot the problem:

• All logs fail to associate with trace IDs. This situation generally stems from several possible causes:

  • The ARMS agent is not mounted.

    Confirm this by checking if all metrics are normally displayed in the ARMS console.

  • The main switch of the agent is disabled or certain plug-ins are disabled. For example, the Tomcat plug-in might be turned off.

    You can go to the application details page and confirm the settings in the Probe switch settings section of the Custom Configurations tab.

  • The server framework is not supported. For instance, unsupported web containers, RPC framework, scheduled task frameworks, and message frameworks are used. For information about the supported frameworks, see Java components and frameworks supported by ARMS.

  • Logging frameworks other than log4j, log4j2, and logback, are used, or deep modifications have been made to these logging frameworks.

  • User log XML configuration errors occurred.

    Follow these troubleshooting steps:

    • If you are using the agent v4.1.6 or later, enable automatic trace ID filling to see if the agent can associate trace IDs. If it does, the issue lies with the user log XML configuration.

    • Introduce ARMS SDK for Java and manually obtain trace IDs when logging messages. Sample code:

      Span span = Tracer.builder().getSpan();   //No new span is created. 
      String traceId = span.getTraceId();
      
      logger.warn("traceId={} this is your log message", traceId)

      If trace IDs can be obtained, this confirms an error in the user log XML configuration.

    • If the preceding verification fails, submit a ticket.

• Some logs fail to associate with trace IDs. This may happen because no trace context exists in the logs.

  Typically, logs printed during general operations do not have a trace context. Only logs printed at request entry points, such as HTTP interfaces, RPC interfaces, scheduled tasks, and message consumption, contain trace contexts.

  Sample scenarios:

  • After an application receives an HTTP request and executes a database query, it prints logs. In this case, a trace context exists, and logs can associate with trace IDs.

  • After receiving an HTTP request, the application submits an asynchronous task through a thread pool to execute a database query and then prints logs.

    • For the ARMS agent v3.x, due to lack of support for asynchronous automatic propagation of trace contexts, the asynchronous threads do not have a trace context, so logs cannot associate with trace IDs.

    • However, for the ARMS agent v4.x, which supports asynchronous automatic propagation of trace contexts, logs can associate with trace IDs.

  • After startup, the application uses the thread pool of the JDK to cyclically execute database queries and print logs after each query. In this scenario, no trace context is available, so logs cannot associate with trace IDs.

  By examining the names of the threads that print logs, you can determine the specific scenario.

  Examples:

  • When using Tomcat as the web container, thread names starting with http-nio- usually represent threads handling HTTP requests, and logs printed by these threads can associate with trace IDs.

  • When using Dubbo as the RPC framework, thread names starting with DubboServerHandler- typically represent threads handling Dubbo requests, and logs printed by these threads can associate with trace IDs.

  If none of the above scenarios apply, submit a ticket.

Why were logs empty when I viewed them from Trace Explorer?

Perform the following steps to troubleshoot the problem:

1. Check whether all logs of the application fail to associate with trace IDs. If all logs of the application have failed to associate with trace IDs, troubleshoot the problem by referring to Why did trace IDs fail to associate with logs after I modified the XML configurations? If not all logs fail to associate with trace IDs, proceed to the next step.

2. Identify interfaces with missing logs for specific trace IDs. If the interfaces that cannot query logs for specific trace IDs are consistently one or a few specific interfaces, it generally indicates that these interfaces themselves do not output logs internally.

3. If none of the above scenarios apply, submit a ticket.

Why did the Logstore have no logs after binding it and the project?

You need to follow the SLS documentation to configure the collection of application logs into the Logstore on your own. ARMS does not automatically collect logs into the Logstore on behalf of you.

Why did the application logs fail to associate with trace IDs, whereas business application logs succeeded?

The agent v3.x has deficiencies in instrumenting Spring Cloud Gateway. We recommend that you upgrade the agent to v4.x.

Why did the logs in an asynchronous task fail to associate with trace IDs after I used TraceCallable.wrap() to encapsulate it?

Upgrade the agent to v4.x. In asynchronous scenarios, no code modification is required.

Why did trace ID association fail after I enabled automatic trace ID filling?

The problem may happen in the following situations:

• All logs failed to associate with trace IDs: In this case, the trace ID filling configuration failed to be pushed to the agent. Check the connectivity of the Application Configuration Management (ACM) server. For more information, see Network configurations required by the ARMS agent for Java.

• Some logs failed to associate with trace IDs: Troubleshoot the problem by referring to Why did trace IDs fail to associate with logs after I modified the XML configurations?