This topic describes how to ship log data from Log Service to TSDB. The timestamp of a log entry in Log Service is mapped to a timestamp in a data entry in TSBD. This facilitates time-series data storage and satisfy different business requirements.

Background information

Logs are essential data that you can use to process historical data, troubleshoot errors, and monitor system activities. In addition, logs are necesssary data sources for data analysts, developers, and O&M personnel.

Log Service allows you to collect log data from multiple cloud services and transform the data based on your business requirements. You can also ship log data from Log Service to TSDB. Timestamps in Log Service are mapped to timestamps in TSDB. This way, log data that is shipped to TSDB is stored as time-series data.

Prerequisites

  • Log Service and TSDB are activated.
  • The source Log Service project and the destination TSDB instance reside in the same region.

Create a log shipping task

The following procedure describes how to create a log shipping task.

  1. Log on to the Log Service console. On the page that appears, click the source project. On the page that appears, click the Logstores tab.
  2. On the Logstores tab, click the > icon of the source Logstore, and then choose Data Transformation > Export > TSDB.
    Create
  3. Click the plus sign (+) next to TSDB. You are redirected to the task configuration page.
    Note If the data in the source Logstore is not transformed, you are prompted to transform the data before shipping. If the data meets the format requirements, you can skip the data transformation and ship the data to TSDB.
  4. Complete parameter configurations.

    The following table describes the parameters in the preceding figure.

    Parameter Description
    Shipping Name The name of the log shipping task.
    Shipping Description The description of the log shipping task. We recommend that you enter an informative description for easy management.
    TSDB Instance The destination TSDB instance.
    Metric Name The metric to which the shipped log data belongs. If no metric is available in a TSDB instance, a metric is automatically created when you ship log data to the instance.
    AccessKeyID The AccessKey ID of the current Alibaba Cloud account that you use to access the destination TSDB instance.
    AccessKeySecret The AccessKey secret of the current Alibaba Cloud account that you use to access the destination TSDB instance.
    Time Point Mapping
    Maps the timestamp in a log entry to the timestamp of a metric value. By default, the log generation time is used as the timestamp of a metric value in TSDB. The default value of this parameter is in the ${logTag:time} format. You can also specify a field in a log entry and map the value of the field to the timestamp of a metric value. In this case, the value of this parameter is in the ${logColumn:xx} format. If the value of the specified field fails to replace the log generation time, the log generation time is used as the timestamp. If you specify a field, you must specify a field whose value can be converted to a UNIX timestamp in seconds or milliseconds. The timestamp of a log entry in Log Service is accurate to the second. You can specify a field if multiple log entries are generated in the same second. You can also use the following expression to specify a contextual field of a log entry:
    ${context:variable}
    For more information about the contextual information of a log entry, see Context query.

    logColumn in the ${logColumn:xx} expression indicates that the value of the specified field in a log entry is mapped to a timestamp in TSDB.

    logTag in the ${logTag:xx} expression indicates that the value of the specified tag field in a log entry is mapped to a timestamp in TSDB. Common tag fields:
    • __topic__: the topic of a log group that consists of multiple log entries. If you specify this tag field, the expression is ${logTag:__topic__}.
    • __source__: the IP address of the server that generates the log entry. If you specify this tag field, the expression is ${logTag:__source__}.
    • __time__: the timestamp of the log entry, accurate to the second. If you specify this tag field, the expression is ${logTag:__time__}.
    Field Mapping Maps log fields to fields in TSDB. If only one field can be mapped, log data is shipped to TSDB in the single-value model. If multiple fields can be mapped, log data is shipped to TSDB in the multi-value model.
    Tag Mapping Maps log fields to tags of a metric. Enter the tag name in the first text box and the tag value in the second text box. You can use the following expression to specify a contextual field:
    ${context:variable}

    For more information about the contextual information of a log entry, see Context query.

    Shipping Time The log generation time of the first log entry to be shipped.
  5. Click Submit.

Related operations

  • After a log shipping task is started, you can click the task name in the console and stop the task or view the status of the task on the task management page.
  • You can also modify the configurations of the task. For information about configuration parameters, see Create a log shipping task.