how to convert logs to traces - Simple Log Service - Alibaba Cloud Documentation Center

You can use the processor_otel_trace plug-in to convert logs collected by Logtail into SLS traces. This topic describes the parameters of the processor_otel_trace plug-in and provides configuration examples.

Entry point

To use a Logtail plugin to process logs, you can add a plugin configuration when you create or modify a Logtail collection configuration. For more information, see Overview of Logtail plugins for data processing.

Configuration description

Important

The processor_otel_trace plug-in is supported only in Logtail versions 1.7.1 and later.

Form configuration

Set Processor Type to Log To Trace. The following table describes the parameters.

Parameters

Parameter	Description
SourceKey	The name of the source field.
Format	The format of the converted data. Valid values: protobuf json protojson
NoKeyError	Specifies whether to report an error if the source field does not exist in the log. The default value is false.
TraceIDNeedDecode	Specifies whether to Base64-decode the trace ID. The default value is false. If you set Format to protojson and the trace ID is Base64-encoded, set this parameter to true. Otherwise, the conversion fails.
SpanIDNeedDecode	Specifies whether to Base64-decode the span ID. The default value is false. If you set Format to protojson and the span ID is Base64-encoded, set this parameter to true. Otherwise, the conversion fails.
ParentSpanIDNeedDecode	Specifies whether to Base64-decode the parent span ID. The default value is false. If you set Format to protojson and the parent span ID is Base64-encoded, set this parameter to true. Otherwise, the conversion fails.

Example

Raw log

INFO OtlpJsonLoggingSpanExporter - {"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"io.opentelemetry.spring-webmvc-3.1","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"eecaa7e1b63c9f61","parentSpanId":"fa85fcee7fe0e0e8","name":"HelloWorldController.sayHello","kind":1,"startTimeUnixNano":"1696659118615748056","endTimeUnixNano":"1696659128701715998","attributes":[{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}}],"events":[],"links":[],"status":{}}]},{"scope":{"name":"io.opentelemetry.tomcat-7.0","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"fa85fcee7fe0e0e8","name":"GET /hello","kind":2,"startTimeUnixNano":"1696659118479360000","endTimeUnixNano":"1696659128703706165","attributes":[{"key":"http.status_code","value":{"intValue":"200"}},{"key":"user_agent.original","value":{"stringValue":"curl/7.84.0"}},{"key":"net.host.name","value":{"stringValue":"localhost"}},{"key":"http.response_content_length","value":{"intValue":"12"}},{"key":"http.target","value":{"stringValue":"/hello"}},{"key":"net.sock.peer.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.host.port","value":{"intValue":"18080"}},{"key":"net.sock.peer.port","value":{"intValue":"53649"}},{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}},{"key":"http.route","value":{"stringValue":"/hello"}},{"key":"net.sock.host.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.protocol.name","value":{"stringValue":"http"}},{"key":"net.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"http.method","value":{"stringValue":"GET"}}],"events":[],"links":[],"status":{}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.19.0"}

Logtail plug-in configuration for data processing
Processing result

JSON configuration

Set type to processor_otel_trace. The following table describes the parameters in the detail object.

Parameters

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the source field.
Format	String	Yes	The format of the converted data. Valid values: protobuf, json, and protojson.
NoKeyError	Boolean	No	Specifies whether to report an error if the source field does not exist in the log. The default value is false.
TraceIDNeedDecode	Boolean	No	Specifies whether to Base64-decode the trace ID. The default value is false. If you set Format to protojson and the trace ID is Base64-encoded, set this parameter to true. Otherwise, the conversion fails.
SpanIDNeedDecode	Boolean	No	Specifies whether to Base64-decode the span ID. The default value is false. If you set Format to protojson and the span ID is Base64-encoded, set this parameter to true. Otherwise, the conversion fails.
ParentSpanIDNeedDecode	Boolean	No	Specifies whether to Base64-decode the parent span ID. The default value is false. If you set Format to protojson and the parent span ID is Base64-encoded, set this parameter to true. Otherwise, the conversion fails.

Example

Raw log

INFO OtlpJsonLoggingSpanExporter - {"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"io.opentelemetry.spring-webmvc-3.1","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"eecaa7e1b63c9f61","parentSpanId":"fa85fcee7fe0e0e8","name":"HelloWorldController.sayHello","kind":1,"startTimeUnixNano":"1696659118615748056","endTimeUnixNano":"1696659128701715998","attributes":[{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}}],"events":[],"links":[],"status":{}}]},{"scope":{"name":"io.opentelemetry.tomcat-7.0","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"fa85fcee7fe0e0e8","name":"GET /hello","kind":2,"startTimeUnixNano":"1696659118479360000","endTimeUnixNano":"1696659128703706165","attributes":[{"key":"http.status_code","value":{"intValue":"200"}},{"key":"user_agent.original","value":{"stringValue":"curl/7.84.0"}},{"key":"net.host.name","value":{"stringValue":"localhost"}},{"key":"http.response_content_length","value":{"intValue":"12"}},{"key":"http.target","value":{"stringValue":"/hello"}},{"key":"net.sock.peer.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.host.port","value":{"intValue":"18080"}},{"key":"net.sock.peer.port","value":{"intValue":"53649"}},{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}},{"key":"http.route","value":{"stringValue":"/hello"}},{"key":"net.sock.host.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.protocol.name","value":{"stringValue":"http"}},{"key":"net.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"http.method","value":{"stringValue":"GET"}}],"events":[],"links":[],"status":{}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.19.0"}

Logtail plug-in configuration for data processing

{
  "processors": [
    {
      "type": "processor_split_log_regex",
      "detail": {
        "PreserveOthers": true,
        "SplitKey": "content",
        "SplitRegex": "INFO\\sOtlpJsonLoggingSpanExporter\\s.+"
      }
    },
    {
      "type": "processor_regex",
      "detail": {
        "SourceKey": "content",
        "Regex": "INFO\\sOtlpJsonLoggingSpanExporter\\s-(\\s.+)",
        "Keys": [
          "trace_data"
        ]
      }
    },
    {
      "detail": {
        "SourceKey": "trace_data",
        "Format": "protojson",
        "NoKeyError": true
      },
      "type": "processor_otel_trace"
    }
  ]
}

Processing result

References

Configure a Logtail pipeline using API operations:
Configure a processor plugin in the console:
- Collect text logs from a host
- Collect container logs from a cluster in the console (stdout/file)
Collect container logs from a Kubernetes cluster using a CRD (stdout/file)