TSDB Writer allows you to write data points to Time Series Database (TSDB) databases that are developed by Alibaba. This topic describes the data types and parameters that TSDB Writer supports and how to configure it by using the code editor.

Notice TSDB Writer supports only exclusive resource groups for Data Integration, but not the default resource group or custom resource groups. For more information, see Use exclusive resource groups for data integration and Add a custom resource group.

Background information

TSDB is a high-performance, cost-effective, stable, and reliable online database service. It features high read/write performance and high compression ratio for data storage. It also enables the interpolation and aggregation of time series data. TSDB is applied in various scenarios, such as the device monitoring system of IoT, energy management system (EMS) for enterprises, security monitoring system in production, and electricity detection system.

TSDB provides you with the capability of writing millions of data points within seconds, together with the benefits of high compression ratio, low-cost data storage, downsampling, interpolation, multi-dimensional aggregation, and visualized query results. These features help you resolve issues such as high storage cost and low writing and query efficiency caused by large amounts of data-collecting points on devices and high frequency of data collection.

TSDB Writer connects to a TSDB instance by using HTTP and writes data points by using the /api/put operation.

Limits

  • TSDB Writer supports only TSDB 2.4.x and later.
  • You can configure TSDB Writer only in the code editor. For more information about TSDB, see What is TSDB.

Data types

Category Data Integration data type TSDB data type
String STRING TIMESTAMP, METRIC, TAGS, and VALUE

Parameters

Source data store Parameter Description Required Default value
TSDB and RDB sourceDbType The type of the source data store. No TSDB
Note The valid values include TSDB and RDB. TSDB refers to OpenTSDB, InfluxDB, Prometheus, and TimeScale databases. RDB refers to relational databases such as MySQL, Oracle, PostgreSQL, and Distributed Relational Database Service (DRDS) databases.
TSDB endpoint The endpoint for connecting to the destination TSDB database by using HTTP. Yes. The endpoint is in the http://IP:Port format. N/A
batchSize The number of data records to write at a time. No. The data type is INT and the value must be greater than 0. 100
maxRetryTime The maximum number of retries allowed after a failure. No. The data type is INT and the value must be greater than 1. 3
ignoreWriteError Specifies whether to ignore write errors. If you set this parameter to true, TSDB Writer continues to perform the write operation when a write error occurs. If the write operation fails after multiple retries, the sync node ends. No. The data type is BOOLEAN. false
RDB endpoint The endpoint for connecting to the destination TSDB database by using HTTP. Yes. The endpoint is in the http://IP:Port format. N/A
column The names of fields in the relational database. Yes N/A
Note The order of the fields specified here must be the same as the order of the fields configured in TSDB Reader.
columnType The types of the fields in the destination TSDB database. The following types of the fields are supported:
  • timestamp: a timestamp.
  • tag: a tag.
  • metric_num: a metric whose value is a number.
  • metric_string: a metric whose value is a string.
Yes N/A
Note The order of the fields specified here must be the same as the order of the fields configured in TSDB Reader.
batchSize The number of data records to write at a time. No. The data type is INT and the value must be greater than 0. 100

Configure TSDB Writer by using the codeless UI

The codeless user interface (UI) is not supported for TSDB Writer.

Configure TSDB Writer by using the code editor

The following example shows how to configure a sync node to write data to a TSDB database. For more information, see Create a sync node by using the code editor.
{
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "concurrent": 1,
            "throttle": true
        }
    },
    "steps": [
        {
            "category": "reader",
            "name": "Reader",
            "parameter": {},
            "stepType": ""
        },
        {
            "category": "writer",
            "name": "Writer",
            "parameter": {
                "endpoint": "http://localhost:8242",
                "sourceDbType": "RDB",
                "batchSize": 256,
                "column": [
                    "name",
                    "type",
                    "create_time",
                    "price"
                ],
                "columnType": [
                    "tag",
                    "tag",
                    "timestamp",
                    "metric_num"
                ]
            },
            "stepType": "tsdb"
        }
    ],
    "type": "job",
    "version": "2.0"
}

Performance test report

  • Characteristics of test data
    • Metric: a metric, which is m.
    • tag_k and tag_v: the keys and values. The first four rows constitute a timeline of 2,000,000 data points, which is equivalent to 10 (zones) × 20 (clusters) × 100 (groups) × 100 (apps). The ip in the last row corresponds to the index of the 2,000,000 data points, starting from 1.
      tag_k tag_v
      zone z1~z10
      cluster c1~c20
      group g1~100
      app a1~a100
      ip ip1~ip2,000,000
    • value: a random value from 1 to 100.
    • interval: a collection period of 10 seconds. If data collection lasts for 3 hours, a total of 2,160,000,000 data points, that is, 3 × 60 × 60/10 × 2,000,000, are collected.
  • Performance test results
    Number of channels Data integration speed (records per second) Data integration bandwidth (Mbit/s)
    1 129,753 15.45
    2 284,953 33.70
    3 385,868 45.71