If you want to monitor the metric change trend of a log field, you can use the e_to_metric function to convert the log field to a metric. Then, you can view the change trend of the metric in a Metricstore. This topic describes how to convert logs to metrics. NGINX access logs are used as an example.
Prerequisites
Logs are collected. For more information, see Data collection overview.
Background information
The operational data of applications includes log data, trace data, and metric data. A log indicates a discrete event. A trace indicates an event that contains a call chain. A metric indicates an event that contains numeric measurements. Logs, traces, and metrics are events. A system that can store events can also store the three types of data. Simple Log Service provides the following two types of stores for data storage:
Logstore
A Logstore in Simple Log Service is a unit that is used to collect, store, and query logs. For more information, see Logstore.
Metricstore
A Metricstore in Simple Log Service is a unit that is used to collect, store, and query metrics. Metricstores are optimized based on metrics. You can use the PromQL syntax that is provided by Prometheus to query data. For more information, see Metricstore.
For most applications, a log contains more information than a metric. Metrics can be considered as logs in a specific format. Simple Log Service allows you to convert logs to metrics. You can use one of the following methods to convert logs to metrics:
Use the Scheduled SQL feature of Simple Log Service to aggregate logs to generate metrics. For more information, see Process and store data from a Logstore to a Metricstore.
Use the e_to_metric function to convert logs to metrics.
Scenarios
For example, you have created a Logstore named nginx-demo in the China (Hangzhou) region to store NGINX access logs.
You need to monitor the changes of the request time (request_time) and response time (upstream_response_time) of each backend server (host), and then visualize the change trend on a dashboard.
body_bytes_sent:1750
host:www.example.com
http_referer:www.guide.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.113.10
remote_addr:203.0.113.10
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
status:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66
To meet the preceding requirements, you must convert the request_time and upstream_response_time fields in logs to metrics, and then add the host label to the metrics.
Step 1: Create a Metricstore
Create a Metricstore named service-metric to store the time series data that is returned during data transformation.
Log on to the Log Service console.
In the Projects section, click the project that you want to manage.
- On the tab, click the + icon.
- In the Create Metricstore panel, configure the following parameters and click OK.
Parameter Description Metricstore Name The name of the Metricstore. The name must be unique in the project to which the Metricstore belongs. After the Metricstore is created, you cannot change its name. Permanent Storage If you turn on Permanent Storage, Log Service permanently stores the collected metrics. Note If you query the data retention period by calling an SDK and the returned result is 3650, metrics are permanently stored.Data Retention Period The retention period of metrics in the Metricstore. Valid values: 15 to 3000. Unit: days. Warning Metrics are automatically deleted after the specified retention period ends.You can configure the Data Retention Period parameter only if you do not turn on Permanent Storage.
Important If you shorten the data retention period, Log Service deletes all expired metrics within 1 hour. The data volume that is displayed for Storage Size(Log) on the homepage of the Log Service console is updated the next day. For example, if you change the value of the Data Retention Period parameter from 5 to 1, Log Service deletes the metrics of the previous four days within 1 hour.Shards The number of shards. Log Service provides shards that allow you to read and write data. Each shard supports a write capacity of 5 MB/s and 500 writes/s and a read capacity of 10 MB/s and 100 reads/s. You can create up to 10 shards in each Metricstore. You can create up to 200 shards in each project. For more information, see Shard. Automatic Sharding If you turn on Automatic Sharding, Log Service increases the number of shards when the existing shards cannot accommodate the data that is written. For more information, see Manage shards. Maximum Shards If you turn on Automatic Sharding, you must configure this parameter to specify the maximum number of readwrite shards that can be created. Maximum value: 256.
Step 2: Create a data transformation job
Use the e_to_metric function to create a data transformation job and store transformed data in the service-metric Metricstore that you created in Step 1.
Go to the data transformation page.
In the Projects section, click the project that you want to manage.
On the tab, click the Logstore that you want to manage.
On the Search & Analysis page, click Data Transformation .
In the upper-right corner of the page, specify a time range for the required log data.
Make sure that log data exists on the Raw Logs tab.
In the code editor, enter a data transformation statement.
Change the name of the request_time field to RequestTime, change the name of the upstream_response_time field to ResponseTime, and then add the host label.
e_to_metric( names=[("request_time", "RequestTime"), ("upstream_response_time", "ResponseTime")], labels=[("host", "hostname")], )
For more information, see e_to_metric.
Click Preview Data.
Create a data transformation job.
Click Save as Transformation Job.
In the Create Data Transformation Job panel, configure the parameters and click OK. The following table describes the parameters.
Parameter
Description
Job Name
The name of the data transformation job. Example: log2mectric.
Authorization Method
Select Default Role to read data from the source Logstore.
Storage Target
Target Name
The name of the storage target. Example: log2mectric.
Target Region
The region where the destination project resides. Example: China (Hangzhou).
Target Project
The name of the destination project to which transformed data is saved.
Target Logstore
The name of the destination Metricstore to which transformed data is saved. Example: service-metric.
Authorization Method
Select Default Role to write transformed data to the destination service-metric Metricstore.
Processing Range
Time Range
Select All.
In the Result dialog box, click Confirm.
After you perform the preceding steps, Simple Log Service transforms the log data in the source Logstore and writes transformed data to the service-metric Metricstore.
Step 3: Query time series data
In the left-side navigation pane, choose .
On the Metricstore tab, select the service-metric Metricstore.
In the upper-right corner of the page, click 15 Minutes (Relative) to specify a time range for the query and analysis.
You can select a relative time or a time frame. You can also specify a custom time range.
NoteThe query and analysis results may contain time series data that is generated 1 minute earlier or later than the specified time range.
On the Query Statements tab, select the RequestTime metric or the ReponseTime metric from the Metrics drop-down list and click Preview.
The following figure shows the change trend of the RequestTime metric for each host.
The following figure shows the change trend of the ReponseTime metric for each host.