TSDB Writer writes data points to Time Series Database (TSDB) that is developed by Alibaba Cloud. This topic describes the data types and parameters that are supported by TSDB Writer and how to configure TSDB Writer by using the codeless user interface (UI) and code editor.

Notice TSDB Writer supports only exclusive resource groups for Data Integration, but not the shared resource group or custom resource groups for Data Integration. For more information, see Create and use an exclusive resource group for Data Integration and Create a custom resource group for Data Integration.

Background information

TSDB is a high-performance, cost-effective, stable, and reliable online time series database service. TSDB features high read and write performance and provides a high compression ratio for data storage. TSDB also enables the interpolation and aggregation of time series data. TSDB can be used in various systems, such as IoT device monitoring systems, energy management systems (EMS) for enterprises, security monitoring systems for production, and electricity consumption monitoring systems.

You can write millions of data points to TSDB within seconds. TSDB provides the following features: high compression ratio, low-cost data storage, downsampling, interpolation, multi-dimensional aggregation, and visualized query results. These features help you resolve issues that are caused by a large number of data collection points on devices and frequent data collection. The issues include high storage costs and low write and query efficiency.

TSDB Writer connects to a TSDB instance by sending an HTTP request and writes data points by using the /api/put HTTP API endpoint.

Limits

  • TSDB Writer supports TSDB V2.4.X and later.
  • You can configure TSDB Writer only in the code editor. For more information about TSDB, see What is TSDB.

Data types

Category Data Integration data type TSDB data type
String STRING String to which a data point in TSDB is serialized. The data point can be a timestamp, metric, tag, or value.

Parameters

Parameter type Parameter Description Required Default value
Common parameter sourceDbType The type of the destination database. No TSDB
Note Valid values: TSDB and RDB. The value TSDB indicates that the destination database is an OpenTSDB, Prometheus, or Timescale database. The value RDB indicates that the destination database is a relational database, such as a MySQL, Oracle, PostgreSQL, or Distributed Relational Database Service (DRDS) database.
Parameters for TSDB endpoint The HTTP endpoint of the destination TSDB database. Specify the endpoint in the format of http://IP address:Port number. Yes No default value
batchSize The number of data records to write at a time. The value of this parameter is of the INT type and must be greater than 0. No 100
maxRetryTime The maximum number of retries allowed after a failure. The value of this parameter is of the INT type and must be greater than 1. No 3
ignoreWriteError Specifies whether to ignore write errors. The value of this parameter is of the BOOLEAN type. If you set this parameter to true, TSDB Writer continues to perform the write operation after a write error occurs. If the write operation fails after the specified number of retries, the synchronization node is terminated. No false
Parameters for RDB endpoint The HTTP endpoint of the destination relational database. Specify the endpoint in the format of http://IP address:Port number. Yes No default value
column The names of the columns to which you want to write data. Yes No default value
Note You must specify the columns in the same order as the columns specified for a reader.
columnType The types of the columns in the relational database. The following types are supported:
  • timestamp: a timestamp column.
  • tag: a tag column.
  • metric_num: a metric column whose value is of a numeric data type.
  • metric_string: a metric column whose value is of a string data type.
Yes No default value
Note You must specify the columns in the same order as the columns specified for a reader.
batchSize The number of data records to write at a time. The value of this parameter is of the INT type and must be greater than 0. No 100

Configure TSDB Writer by using the codeless UI

This method is not supported.

Configure TSDB Writer by using the code editor

In the following code, a synchronization node is configured to write data to a TSDB database. For more information about how to configure a synchronization node by using the code editor, see Create a synchronization node by using the code editor. 
{
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "steps": [
        {
            "category": "reader",
            "name": "Reader",
            "parameter": {},
            "stepType": ""
        },
        {
            "category": "writer",
            "name": "Writer",
            "parameter": {
                "endpoint": "http://localhost:8242",
                "sourceDbType": "RDB",
                "batchSize": 256,
                "column": [
                    "name",
                    "type",
                    "create_time",
                    "price"
                ],
                "columnType": [
                    "tag",
                    "tag",
                    "timestamp",
                    "metric_num"
                ]
            },
            "stepType": "tsdb"
        }
    ],
    "type": "job",
    "version": "2.0"
}

Performance test report

  • Characteristics of test data
    • Metric: a metric, which is m.
    • tag_k and tag_v: the key and value of a tag. The keys and values of the first four tags constitute a time series of 2,000,000 data points. The number of data points is calculated by using the following formula: 10 (zones) × 20 (clusters) × 100 (groups) × 100 (applications). The ip tag corresponds to the index of the 2,000,000 data points, starting from 1.
      tag_k tag_v
      zone z1 to z10
      cluster c1 to c20
      group g1 to g100
      app a1 to a100
      ip ip1 to ip2,000,000
    • value: a random value from 1 to 100.
    • interval: a collection interval of 10 seconds. The total duration of data collection is 3 hours, and a total number of 2,160,000,000 data points are collected. The number of data points is calculated by using the following formula: 3 × 60 × 60/10 × 2,000,000.
  • Performance test results
    Number of channels Data integration speed (record/s) Data integration bandwidth (Mbit/s)
    1 129,753 15.45
    2 284,953 33.70
    3 385,868 45.71