All Products
Search
Document Center

Simple Log Service:Convert logs to metrics

Last Updated:Aug 29, 2023

If you want to monitor the metric change trend of a log field, you can use the e_to_metric function to convert the log field to a metric. Then, you can view the change trend of the metric in a Metricstore. This topic describes how to convert logs to metrics. NGINX access logs are used as an example.

Prerequisites

Logs are collected. For more information, see Data collection overview.

Background information

The operational data of applications includes log data, trace data, and metric data. A log indicates a discrete event. A trace indicates an event that contains a call chain. A metric indicates an event that contains numeric measurements. Logs, traces, and metrics are events. A system that can store events can also store the three types of data. Simple Log Service provides the following two types of stores for data storage:

  • Logstore

    A Logstore in Simple Log Service is a unit that is used to collect, store, and query logs. For more information, see Logstore.

  • Metricstore

    A Metricstore in Simple Log Service is a unit that is used to collect, store, and query metrics. Metricstores are optimized based on metrics. You can use the PromQL syntax that is provided by Prometheus to query data. For more information, see Metricstore.

For most applications, a log contains more information than a metric. Metrics can be considered as logs in a specific format. Simple Log Service allows you to convert logs to metrics. You can use one of the following methods to convert logs to metrics:

Scenarios

For example, you have created a Logstore named nginx-demo in the China (Hangzhou) region to store NGINX access logs.

You need to monitor the changes of the request time (request_time) and response time (upstream_response_time) of each backend server (host), and then visualize the change trend on a dashboard.

body_bytes_sent:1750
host:www.example.com
http_referer:www.guide.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.113.10
remote_addr:203.0.113.10
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
status:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

To meet the preceding requirements, you must convert the request_time and upstream_response_time fields in logs to metrics, and then add the host label to the metrics.

Step 1: Create a Metricstore

Create a Metricstore named service-metric to store the time series data that is returned during data transformation.

  1. Log on to the Log Service console.

  2. In the Projects section, click the project that you want to manage.

  3. On the Time Series Storage > Metricstore tab, click the + icon.
  4. In the Create Metricstore panel, configure the following parameters and click OK.
    ParameterDescription
    Metricstore NameThe name of the Metricstore. The name must be unique in the project to which the Metricstore belongs. After the Metricstore is created, you cannot change its name.
    Permanent StorageIf you turn on Permanent Storage, Log Service permanently stores the collected metrics.
    Note If you query the data retention period by calling an SDK and the returned result is 3650, metrics are permanently stored.
    Data Retention PeriodThe retention period of metrics in the Metricstore. Valid values: 15 to 3000. Unit: days.
    Warning Metrics are automatically deleted after the specified retention period ends.

    You can configure the Data Retention Period parameter only if you do not turn on Permanent Storage.

    Important If you shorten the data retention period, Log Service deletes all expired metrics within 1 hour. The data volume that is displayed for Storage Size(Log) on the homepage of the Log Service console is updated the next day. For example, if you change the value of the Data Retention Period parameter from 5 to 1, Log Service deletes the metrics of the previous four days within 1 hour.
    ShardsThe number of shards. Log Service provides shards that allow you to read and write data. Each shard supports a write capacity of 5 MB/s and 500 writes/s and a read capacity of 10 MB/s and 100 reads/s. You can create up to 10 shards in each Metricstore. You can create up to 200 shards in each project. For more information, see Shard.
    Automatic ShardingIf you turn on Automatic Sharding, Log Service increases the number of shards when the existing shards cannot accommodate the data that is written. For more information, see Manage shards.
    Maximum ShardsIf you turn on Automatic Sharding, you must configure this parameter to specify the maximum number of readwrite shards that can be created. Maximum value: 256.

Step 2: Create a data transformation job

Use the e_to_metric function to create a data transformation job and store transformed data in the service-metric Metricstore that you created in Step 1.

  1. Go to the data transformation page.

    1. In the Projects section, click the project that you want to manage.

    2. On the Log Storage > Logstores tab, click the Logstore that you want to manage.

    3. On the Search & Analysis page, click Data Transformation .

  2. In the upper-right corner of the page, specify a time range for the required log data.

    Make sure that log data exists on the Raw Logs tab.

  3. In the code editor, enter a data transformation statement.

    Change the name of the request_time field to RequestTime, change the name of the upstream_response_time field to ResponseTime, and then add the host label.

    e_to_metric(
        names=[("request_time", "RequestTime"), ("upstream_response_time", "ResponseTime")],
        labels=[("host", "hostname")],
    )

    For more information, see e_to_metric.

  4. Click Preview Data.

    Preview data
  5. Create a data transformation job.

    1. Click Save as Transformation Job.

    2. In the Create Data Transformation Job panel, configure the parameters and click OK. The following table describes the parameters.

      Parameter

      Description

      Job Name

      The name of the data transformation job. Example: log2mectric.

      Authorization Method

      Select Default Role to read data from the source Logstore.

      Storage Target

      Target Name

      The name of the storage target. Example: log2mectric.

      Target Region

      The region where the destination project resides. Example: China (Hangzhou).

      Target Project

      The name of the destination project to which transformed data is saved.

      Target Logstore

      The name of the destination Metricstore to which transformed data is saved. Example: service-metric.

      Authorization Method

      Select Default Role to write transformed data to the destination service-metric Metricstore.

      Processing Range

      Time Range

      Select All.

    3. In the Result dialog box, click Confirm.

After you perform the preceding steps, Simple Log Service transforms the log data in the source Logstore and writes transformed data to the service-metric Metricstore.

Step 3: Query time series data

  1. In the left-side navigation pane, choose Time Series Storage > Metricstore .

  2. On the Metricstore tab, select the service-metric Metricstore.

  3. In the upper-right corner of the page, click 15 Minutes (Relative) to specify a time range for the query and analysis.

    You can select a relative time or a time frame. You can also specify a custom time range.

    Note

    The query and analysis results may contain time series data that is generated 1 minute earlier or later than the specified time range.

  4. On the Query Statements tab, select the RequestTime metric or the ReponseTime metric from the Metrics drop-down list and click Preview.

    • The following figure shows the change trend of the RequestTime metric for each host.RequestTime

    • The following figure shows the change trend of the ReponseTime metric for each host.ReponseTime