All Products
Search
Document Center

Create a data export task

Last Updated: May 31, 2020

Overview

Alibaba Cloud Log Service is a one-stop logging service that allows you to collect logs from a variety of cloud products and process the collected logs. Time Series Database (TSDB) allows you to import logs from Log Service and map the log timestamps to TSDB timestamps for time series data storage.

For more information about Log Service, see the Log Service documentation.

Prerequisites

  • The Log Service and Time Series Database (TSDB) services are purchased.
  • The Log Service and TSDB services reside in the same region. Otherwise, they cannot exchange data.

Create a task

When you create a data export task in the Log Service console to export data from Log Service to TSDB, you can specify the task details. For example, you can specify the target TSDB instance, the time of the earliest data record to be exported, the Log Service topic, and the TSDB metric.

This topic describes how to create a data export task for TSDB.

Operations in the Log Service console

  1. Log on to the Log Service console, and select the project and the Logstore from which the data is to be exported.
  2. Click the right arrow for the target Logstore to show the Logstore drop-down list. Then, click Export, click TSDB, and then click the plus sign (+) on the right side of TSDB.

  3. On the page that appears, configure the parameters.

  4. If the data in the Logstore is not processed, a message is displayed to check whether you want to process log data before data shipping. If the log data uses the standard format, you can skip data processing and ship the log data to TSDB.

Parameters

Parameter
Description
Shipper Name The task name, which must be unique.
Shipper Description The task description.
TSDB Instance The TSDB instance to which the data is exported. You can select a TSDB instance from the drop-down list that shows the instances that you have purchased in the current region.
Metric The TSDB metric for which the data is exported.
AccessKey ID The AccessKey ID of the logon account. You must specify the AccessKey ID for identity verification when you export data from Log Service to TSDB.
AccessKey Secret The AccessKey secret of the logon account. You must specify the AccessKey secret for identity verification when you export data from Log Service to TSDB.
Time Point Mapping Specifies the mappings between the timestamps in log records and the TSDB timestamps. The default value of this parameter is ${logTag:__time__}. The default value indicates that the time points when logs were generated are considered as the TSDB timestamps. You can specify an expression for this parameter.
Field Mapping Specifies the mappings between the fields in log records and TSDB fields. In the first text box, enter the name of a field in the log records. In the second text box, enter the name of a TSDB field. If you specify only one field mapping, data is exported based on the single-value data model. In this case, the name of the TSDB field must be value. If you specify multiple field mappings, data is exported based on the multi-value data model.
Tag Mapping Specifies the TSDB tags for the exported data records. In the first text box, enter a tag name. In the second text box, enter a tag value. You can also use expressions to specify the tag names and the tag values.
Shipping Time Specifies the time when the earliest data record to be shipped was generated. Log Service does not ship the data that is generated earlier than the specified time.
  1. If no metrics exist, the system automatically creates a metric.

  2. When Log Service exports logs to TSDB, the setting of the Time Point Mapping parameter determines the Timestamp field values of TSDB data points. The default value of this parameter is ${logTag:time}. The default value indicates that the time points when log records were generated are considered as the TSDB timestamps. The timestamps of log records are specified by the time parameter and use the second time unit. Therefore, if multiple log records occur within one second, you must determine the log records based on which the TSDB timestamps are determined.

  3. You can use the ${logTag:xx} and ${logColumn:xx} expressions to specify the numeric values for the Time Point Mapping parameter. The values of the fields specified by the expressions must be able to be converted into numeric values. Each field value must be converted into the number of seconds or milliseconds that have elapsed since 00:00:00, January 1, 1970.

  4. If the values of the fields specified by the expressions fail to be converted into numeric values, the system uses the log record timestamps as the TSDB timestamps. The log record timestamps are specified by the ${logTag:time} element.

Use expressions to reference the context information

When you configure data export tasks, you can use expressions to reference the context information for the Time Point Mapping and Tag Mapping parameters. The expression syntax is described as follows:

${context:variable}

  • The logColumn prefix in an expression indicates that the expression references the column values in the original logs of Log Service as the target values.
  • The logTag prefix in an expression indicates that the expression references the tag values in the original logs of Log Service as the target values.

You can always find the following tags in the logs of Log Service:

  • __topic__: indicates the service tag of an entire log group. A log group may contain one or more log records. The expression format is ${logTag:__topic__}.
  • __source__: indicates the IP address of the machine that generates the log record. The expression format is ${__source__}.
  • __time__: indicates the timestamp of the log record. The unit of the timestamp is seconds. The expression format is ${__time__}.

For more information about Log Service context variables, see the Log Service documentation.

Submit the task

After you specify the required parameters, click Submit. Now you have created a data export task for TSDB.

Considerations

  • A data export task is instantly started after the task is created. To stop the task or view the task status, you can click the task name in the Log Service console. On the page that appears, you can manage the task based on your business needs.
  • After a data export task is created, you can modify the task configurations based on your business needs. For more information, see the “Modify a data export task” topic.