Associate application logs with trace IDs in Simple Log Service - Managed Service for OpenTelemetry

You can associate application logs with trace IDs in Simple Log Service by using the event configuration feature. When an error occurs, use the trace ID written to your application logs to quickly locate the corresponding trace and troubleshoot the issue.

Prerequisites

The application is connected to Managed Service for OpenTelemetry. For more information, see Integrate services or components.
Application logs are imported to Simple Log Service, and trace IDs or span IDs are written to the logs. For more information, see Collect and analyze ECS text logs using LoongCollector and Logger MDC auto-instrumentation.
Custom business fields, such as trans-id, must be written to the logs by the application layer through an SDK or a logging framework. Simple Log Service does not generate such fields automatically. To use a custom field as a separate field, use a Logtail processor in regex mode to extract it from the log content. Custom fields can only be placed in the message field. System fields such as thread cannot be modified.

Step 1: Extract the trace ID

If the trace ID is stored in a separate field in the logs, skip this step.
If the trace ID is contained in a field such as the content field, extract it and display it in a separate field.

Log on to the Simple Log Service console.
Select the project and Logstore in which the application logs are stored.
Use a Logtail processor to extract the trace ID from the logs in regex mode and display the trace ID in a separate field.
1. In the left-side Logstores panel, choose Data Collection > Logtail Configurations. On the Logtail Configuration page, find the Logtail configuration that you want to manage and click Manage Logtail Configuration in the Actions column.
2. Click Edit to start editing. In the Processor Configurations section on the Configuration Details tab, click Add Processor and select Extract Field (Regex Mode) from the Processor Type drop-down list.
  In this example, the trace ID or span ID is contained in the content field. The following sample shows the content field of a log entry:
```
2024-05-06 12:12:49.685 [http-nio-9190-exec-3] DEBUG io.lettuce.core.protocol.DefaultEndpoint - traceId:ea1a00002d17150191696858089d0007 - [channel=0x5d069239, /10.0.0.45:49614 -> redis/192.168.216.80:6379, epid=0x1] write() done
```
  In the Basic Configuration section, set the Original Field parameter to content, set the Regular Expression parameter to traceId:([a-zA-Z0-9]+), and add the trace ID or span ID to the New Field Name field. For more information, see Data parsing plugins.
  After the configuration is complete, the extracted log entry looks like the following.
```
__tag__:_cluster_id_: xxx
__tag__:_node_ip_: xxx.xxx.xxx.7
__tag__:_node_name_: cn-hangzhou.xxx.xxx.7
_container_ip_: xxx.xxx.xxx 40
_container_name_: adservice
_image_name_: ghcr.io/open-telemetry/demo:1.4.0-adservice
_namespace_: otel-demo
_pod_name_: opentelemetry-demo-adservice-xxx
_pod_uid_: xxx
_source_: stdout
_time_: 2024-05-07T10:36:48.078285414+08:00
content: 2024-05-07 02:36:48 - oteldemo.AdService - Targeted ad request received for [accessories] trace_id=xxx span_id=xxx trace_flags=01
span_id: xxx
traceId: xxx
```

Step 2: Configure an event

Click the trace ID or span ID in a log and then click Go to configuration.
In the Advanced Event Settings dialog box, add an event for the traceId or spanId field and click OK. The following parameters are available.
- Configuration Name: Enter a custom name, such as Start Trace Analysis.
- Event Action: Select Create Custom HTTP URL.
- Protocol: Select Custom and enter https in the input box.
- Enter a URL:
  - To filter traces by trace ID, specify the URL in the format of trace.console.alibabacloud.com/#/${regionId}/tracing-explorer?source=XTRACE&filters=traceId="${traceId}". Replace ${regionId} with the ID of the region where your application is connected to Managed Service for OpenTelemetry.
    For example, if your application is connected to Managed Service for OpenTelemetry in the China (Hangzhou) region, replace ${regionId} with cn-hangzhou. The complete URL is trace.console.alibabacloud.com/#/cn-hangzhou/tracing-explorer?source=XTRACE&filters=traceId="${traceId}".
  - To filter traces by span ID, extract the span ID into a separate field and replace ${traceId} in the URL with ${spanId}. To filter traces by both trace ID and span ID, use the following URL: trace.console.alibabacloud.com/#/cn-hangzhou/tracing-explorer?source=XTRACE&filters=traceId="${traceId}" AND spanId="${spanId}".
  - To filter traces by custom attributes such as Attributes and Resources, specify the URL in the following format: trace.console.alibabacloud.com/#/cn-hangzhou/tracing-explorer?source=XTRACE&filters=${Name of the custom attribute in traces}="${Name of the field in logs}".
    For example, if you want to filter traces by namespace, the logs contain the namespace field, and the k8s.namespace.name attribute is added to the traces, the complete URL is trace.console.alibabacloud.com/#/cn-hangzhou/tracing-explorer?source=XTRACE&filters=resources.k8s.namespace.name="${namespace}".
Click the trace ID or span ID again in the log and click Start Trace Analysis to go to the Trace Explorer page of Managed Service for OpenTelemetry.
On the Trace Explorer page, query the trace by trace ID. For more information, see Trace Explorer.
On the Trace Explorer page, you can use the filter panel on the left to filter traces by status, latency, application name, API name, and host address. The upper-right area displays statistics charts including call counts, HTTP errors, and latency percentiles. The table below lists each trace record with details such as TraceId, API name, application name, latency, status, start time, and host address.