All Products
Search
Document Center

Create a task to export data

Last Updated: May 19, 2022

This topic describes how to create a task to export log records to a Lindorm Time Series Database (TSDB) instance.

Background information

Log Service helps you collect log data from multiple cloud services and process the log data. TSDB allows you to import log data from Log Service. Data in Log Service is mapped to data points in TSDB. This way, the log data that is shipped to TSDB is stored as the time series data.

Prerequisites

  • A Log Service project and a TSDB instance are created.

  • Your Log Service project and TSDB instance are deployed in the same region. Data interaction across regions is not supported.

Procedure

When you create a task to export data to TSDB, you can configure the task in the Log Service console. For example, you can specify the TSDB instance to which you want to export data, the time when the earliest data record that you want to export is generated, and the TSDB topic to which the data is exported.

  1. Log on to the Log Service console and click the name of the project from which you want to export data. On the page that appears, click Logstores.

  2. Expand the drop-down list of a Logstore and choose Data Transformation > Export > TSDB.Log Service console

  3. Click the plus sign (+) next to TSDB. You are redirected to the task configuration page. On the page that appears, create a task to export data.

  4. If the data in the source Logstore is not transformed, the system displays a message that prompts you to transform the data before the data is shipped. If the data meets the format requirements, you can skip data transformation and ship the data to TSDB.

  5. The following table describes the parameters of the task.333

    Parameter

    Description

    Delivery Name

    The unique name of the task.

    Delivery Description

    The description of the task.

    TSDB Instance

    The TSDB instance to which you want to export data. Select a TSDB instance from the drop-down list.

    Metric Name

    The metric to which the data to be exported belongs.

    AccessKeyID

    The AccessKey ID of the Alibaba Cloud account that you use to access the destination TSDB instance.

    AccessKeySecret

    The AccessKey secret of the Alibaba Cloud account that you use to access the destination TSDB instance.

    Time Point Mapping

    The expression that is used to map the timestamps in log records to the timestamps in TSDB. The default value is ${logTag:__time__}. The default value specifies that the points in time when log records are generated are used as the timestamps in TSDB. You can change the expression based on your business requirements.

    Field Mappings

    The mappings between the fields in Log Service and the fields in TSDB. In the text box on the left side of the equal sign (=), enter the name of a field in Log Service. In the text box on the right side of the equal sign (=), enter the name of the corresponding field in TSDB. If you specify only one field mapping, the single-value model is used to export log data to TSDB and the field key must be value. If you specify multiple data mappings, the multi-value model is used to export log data to TSDB.

    Tag Mappings

    The tags of the exported data records in TSDB. In the text box on the left side of the equal sign (=), enter a tag name. In the text box on the right side of the equal sign (=), enter a tag value. You can also use expressions to specify the tag names and values.

    Delivery Time

    The point in time at which the first log entry that you want to ship is generated. The log entries that are generated before the specified point in time are not shipped.

    Note

    If no metrics exist, the system automatically creates a metric. When log records are exported from Log Service to TSDB, the setting of the Time Point Mapping parameter determines the Timestamp field value in TSDB. The default value of the Time Point Mapping parameter is ${logTag:__time__}. The timestamp of a log record in Log Service is accurate to seconds. If multiple log records are generated in the same second, you must specify a field. To change the value of the Time Point Mapping parameter, configure an expression in the ${logTag:xx} or ${logColumn:xx} format. If you need to specify a field, specify a field whose value can be converted into a UNIX timestamp in seconds or milliseconds. Note: If the expression that you specify is invalid, the system uses the default expression ${logTag:__time__}. This expression specifies that the timestamps in log records are used as the timestamps in TSDB.

    Use expressions to reference the context information

    When you configure a data export task, you can specify expressions as the values of the Time Point Mapping parameter and the Tag Mappings parameter to reference the context information. The following expression syntax is used:

    Note

    ${ context:variable }

    • logColumn in the ${logColumn:xx} expression specifies that the value of the specified field in a log record is mapped to a timestamp in TSDB.

    • logTag in the ${logTag:xx} expression specifies that the value of the specified tag field in a log record is mapped to a timestamp in TSDB.

    The following list describes the tags that exist in log records of Log Service:

    • __topic__: the topic of a log group that consists of multiple log records. The expression is in the ${logTag:__topic__} format.

    • __source__: the IP address of the server that generates the log record. The expression is in the ${logTag:__source__} format.

    • __time__: the timestamp in the log record. The unit of the timestamp is seconds. The expression is in the ${logTag:__time__} format.

Submit the task

After you configure the task, click Submit.

What to do next

  • The task is started after you create the task. If you want to suspend the task or view the task status, click the task name in the Log Service console. Then, you can manage the task as needed.

  • After the task is created, you can modify the task configuration. For more information, see Modify a data export task.