how to convert logs to sls metrics - Simple Log Service - Alibaba Cloud Documentation Center

When you collect logs using Logtail, you can use the processor_log_to_sls_metric plug-in to convert the logs into SLS metrics. This topic describes the parameters of the processor_log_to_sls_metric plug-in and provides configuration examples.

Entry point

To use a Logtail plugin to process logs, you can add a plugin configuration when you create or modify a Logtail collection configuration. For more information, see Overview of Logtail plugins for data processing.

Configuration description

Important

The processor_log_to_sls_metric plug-in is supported only in Logtail 1.7.1 and later.

Form-based configuration

Set Processor Type to Log To Metric. The following table describes the parameters.

Parameters

Parameter	Description
MetricTimeKey	The time field in the log. This field is mapped to the `__time_nano__` field in a metric. By default, the value of the `__time__` field in the log is extracted. Ensure that the specified field is a valid, formatted timestamp. Unix timestamps in seconds (10 digits), milliseconds (13 digits), microseconds (16 digits), and nanoseconds (19 digits) are supported.
MetricLabelKeys	A list of keys from the log to be used as labels in the `__labels__` field. The key must match the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`. The value cannot contain vertical bars (\|) or `#$#`. For more information, see Time series data. Do not add the `__labels__` field to the MetricLabelKeys parameter. If the raw log contains an `__labels__` field, its value is appended to the new `__labels__` field.
MetricValues	Specifies the metric name and metric value. The metric name corresponds to the `__name__` field and must match the regular expression `^[a-zA-Z_:][a-zA-Z0-9_:]*$`. The metric value corresponds to the `__value__` field and must be of the Double type. For more information, see Time series data.
CustomMetricLabels	Custom labels to add to the `__labels__` field. The key must match the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`. The value cannot contain vertical bars (\|) or `#$#`. For more information, see Time series data.
IgnoreError	Specifies whether to output an error log if no logs are matched.

Configuration example

Raw log

{
    "script_md5":"8c6aebe9****c27f",
    "remote_addr":"123.XX.XX.123",
    "remote_user":"-",
    "time_local":"15/Aug/2023:12:03:20",
    "method":"GET",
    "url":"/www.a.com",
    "protocol":"HTTP/1.1",
    "status":"404",
    "body_bytes_sent":"4146",
    "http_referer":"-",
    "http_user_agent":"curl/7.74.0",
    "http_x_forwarded_for":"-",
    "__labels__":"field1#$#field1|field2#$#field2|tag1#$#tag1|tag2#$#tag2",
    "body_bytes_sent_name":"body_bytes_sent",
    "status_name":"status",
    "httptime":"1692101000",
    "__time__":"1692101000"
}

Logtail plug-in configuration for data processing

Processing results

The first metric

{
    "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com",
    "__name__":"body_bytes_sent",
    "__value__":"4146",
    "__time_nano__":"1692101000000000000",
    "__time__":"1692101000"
}

The second metric

{
    "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com",
    "__name__":"status",
    "__value__":"404",
    "__time_nano__":"1692101000000000000",
    "__time__":"1692101000"
}

JSON configuration

Set the type parameter to processor_log_to_sls_metric. The following table describes the parameters in the detail object.