All Products
Search
Document Center

Create a data export task

Last Updated: Aug 25, 2021

This topic describes how to create a task to export log records to Time Series Database (TSDB).

Background information

Log Service allows you to collect log data from multiple cloud services and transform the data. You can also ship log data from Log Service to TSDB. Timestamps in Log Service are mapped to timestamps in TSDB. This way, log data that is shipped to TSDB is stored as time-series data.

Prerequisites

  • A Log Service project and a TSDB instance are created.

  • Your Log Service project and TSDB instance are deployed in the same region. Cross-region data interaction is not supported.

Procedure

When you create a task to export data to TSDB, you can configure the task in the Log Service console. For example, you can specify the destination TSDB instance, the time of the earliest data record to be exported, and the TSDB topic to which the data is exported.

  1. Log on to the Log Service console and click the source project. On the page that appears, click the Logstores tab.

  2. Click the right arrow on the right of the source Logstore name to expand the drop-down list, choose Export > TSDB, and then click the plus sign (+).

  3. On the page that appears, configure the task.

  4. If the data in the source Logstore is not transformed, you are prompted to transform the data before the data is shipped. If the data meets the format requirements, you can skip the data transformation and ship the data to TSDB.

  5. The following table describes the parameters of the task.

    Parameter

    Description

    Delivery Name

    The unique name of the task.

    Delivery Description

    The description of the task.

    TSDB Instance

    The name of the destination TSDB instance in the current region.

    Measure Name

    The metric to which the shipped log data belongs. The metric is in TSDB.

    AccessKeyID

    The AccessKey ID of the current Alibaba Cloud account that you use to access the destination TSDB instance.

    AccessKeySecret

    The AccessKey secret of the current Alibaba Cloud account that you use to access the destination TSDB instance.

    Time Point Mapping

    The expression used to map the timestamps in log records to the timestamps in TSDB. The default value is ${logTag:__time__}. The default value specifies that the points in time when logs are generated are used as the timestamps in TSDB. You can change the expression based on your business requirements.

    Field Mappings

    The mappings between the fields in log records and the fields in TSDB. In the text box on the left of the equal sign (=), enter the name of a field in the log records. In the text box on the right of the equal sign (=), enter the name of the corresponding field in TSDB. If you specify only one field mapping, the single-value model is used to ship log data to TSDB and the field key must be value. If you specify multiple data mappings, the multi-value model is used to ship log data to TSDB.

    Tag Mappings

    The TSDB tags of the exported data records. In the text box on the left of the equal sign (=), enter a tag name. In the text box on the right of the equal sign (=), enter a tag value. You can also use expressions to specify the tag names and values.

    Delivery Time

    The generation time of the first log record to be shipped.

    Note

    If no metrics exist, the system automatically creates a metric. When log records are exported from Log Service to TSDB, the setting of the Time Point Mapping parameter determines the Timestamp field values of TSDB. The default value of the Time Point Mapping parameter is ${logTag:__time__}. The timestamp of a log record in Log Service is accurate to the second. If multiple log records are generated in the same second, you must specify a field. To change the value of the Time Point Mapping parameter, configure an expression in the ${logTag:xx} or ${logColumn:xx} format. If you specify a field, you must specify a field whose value can be converted to a UNIX timestamp in seconds or milliseconds. Note: If the expression that you specify is invalid, the system uses the default expression: ${__logTag:time__}. This expression specifies that the timestamps in log records are used as the timestamps in TSDB.

    Use expressions to reference the context information

    When you configure a data export task, you can specify expressions as the values of the Time Point Mapping parameter and the Tag Mappings parameter to reference the context information. The following expression syntax is used:

    Note

    ${ context:variable }

    • logColumn in the ${logColumn:xx} expression specifies that the value of the specified field in a log record is mapped to a timestamp in TSDB.

    • logTag in the ${logTag:xx} expression specifies that the value of the specified tag field in a log record is mapped to a timestamp in TSDB.

    The following list describes the tags that always exist in log records of Log Service:

    • __topic__: the topic of a log group that consists of multiple log records. The expression is in the ${logTag:__topic__} format.

    • __source__: the IP address of the server that generates the log record. The expression is in the ${logTag:__source__} format.

    • __time__: the timestamp in the log record. The unit of the timestamp is seconds. The expression is in the ${logTag:__time__} format.

Submit the task

After you configure the task, click Submit.

Note

  • The task is immediately started after it is created. If you want to suspend the task or view the task status, click the task name in the Log Service console. Then, you can manage the task based on your business requirements.

  • After the task is created, you can modify the configuration. For more information, see the "Modify a data export task" topic.