TSDB Reader allows you to read data from Time Series Database (TSDB). This topic describes the data types and parameters supported by TSDB Reader and how to configure it by using the code editor.

Notice Currently, you can only use exclusive resource groups for Data Integration for TSDB Reader. The default resource group and custom resource groups are not supported. For more information, see Use exclusive resource groups for data integration and Add a custom resource group.

Background

TSDB is a high-performance, cost-effective, stable, and reliable online time series database service. It features high read/write performance and high compression ratio for data storage. It also enables the interpolation and aggregation of time series data. TSDB is widely applied in Internet of Things (IoT) and Internet. It can be used to monitor devices and business and send alerts in real time. For more information, see What is TSDB.

TSDB Reader connects to a TSDB instance through an HTTP request and obtains data points through the /api/query or /api/mquery HTTP API operation. For more information, see HTTP API overview. A sync node is split to multiple export tasks based on the time series and time range.

Limits

  • If a metric reports a data volume larger than the specified threshold for an hour, you need to change the value of the -j parameter to adjust the memory size of the Java virtual machine (JVM).
    If the synchronization speed of the downstream writer is lower than that of TSDB Reader, the task backlog may occur. You need to change the memory size of the JVM to an appropriate value. For example, run the following command to extract data from an Alibaba Cloud TSDB database to a local data store with the JVM memory size specified:
    python datax/bin/datax.py tsdb2stream.json -j "-Xms4096m -Xmx4096m"
  • The specified start time and end time are automatically converted to on-the-hour time.

    For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).

Data types

The following table lists the data types supported by TSDB Reader.
Data Integration data type TSDB data type
STRING String to which a data point in TSDB is serialized, including the timestamp, metric, tags, fields, and value

Parameters

Parameter Description Required Default value
name The name of the reader. Yes tsdbreader
sinkDbType The type of the destination database.
Valid values: TSDB and RDB.
  • A value of TSDB indicates that the destination data store is an Alibaba Cloud TSDB, OpenTSDB, InfluxDB, Prometheus, or Timescale database.
  • A value of RDB indicates that the destination data store is a relational database such as an AnalyticDB for MySQL, Oracle, MySQL, PostgreSQL, or Distributed Relational Database Service (DRDS) database.
No TSDB
endpoint The HTTP endpoint for connecting to the source TSDB database. Specify the endpoint in the http://IP address:Port format. Yes None
column The columns to be synchronized. The value of this parameter varies with the type of the destination database.
  • TSDB: The metrics to be synchronized.
  • RDB: The tags to be synchronized and the __metric__, __ts__, and __value__ fields.

    The __metric__ field is added to synchronize metrics and the __ts__ field is added to synchronize timestamps. If you want to synchronize single-value data, the __value__ field is added to synchronize values. To synchronize multi-value data, specify the field parameter.

Yes None
metric The metrics to be synchronized. The parameter is valid only when the destination data store is a relational database. No None
field The fields to be synchronized. This parameter is valid only for multi-value data synchronization. No None
tag The tags to be synchronized, in the format of key-value pairs. These tags are used to filter time series. No None
splitIntervalMs The interval of collecting data to be synchronized. An export task is split to multiple subtasks to collect data based on the interval. Unit: milliseconds. Yes None
beginDateTime The start time of the time range of the data points to be synchronized, in the format of yyyy-MM-dd HH:mm:ss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be synchronized.
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
Yes None
endDateTime The end time of the time range of the data points to be synchronized, in the format of yyyy-MM-dd HH:mm:ss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be synchronized.
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
Yes None

Configure TSDB Reader by using the codeless UI

Currently, the codeless user interface (UI) is not supported for TSDB Reader.

Configure TSDB Reader by using the code editor

For more information about how to use the code editor, see Create a sync node by using the code editor.
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write time series data to a local data store.
    Sample time series data:
    {"metric":"m","tags":{"app":"a19","cluster":"c5","group":"g10","ip":"i999","zone":"z1"},"timestamp":1546272263,"value":1}
    Sample code:
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "TSDB", // The type of the destination database. 
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "m"
                    ],
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"stream",
                "parameter":{
    
                },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write relational data to a local data store.
    Sample relational data:
    m 1546272125 a1    c1 g2 i3021 z4 1.0
    Sample code:
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB", // The type of the destination database. 
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "__value__"
                    ],
                    "metric": [
                      "m"
                    ],
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"stream",
                "parameter":{
    
                },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write single-value data to an AnalyticDB for MySQL database.
    
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "__value__"
                    ],
                    "metric": [
                      "m"
                    ],
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"stream",
                "parameter":{
    
                },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write multi-value data to an AnalyticDB for MySQL database.
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "load",
                      "memory",
                      "cpu"
                    ],
                    "metric": [
                      "m_field"
                    ],
                    "field": {
                      "m_field": [
                        "load",
                        "memory",
                        "cpu"
                      ]
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"ads",
                  "parameter": {
                    "username": "******",
                    "password": "******",
                    "column": [
                      "`metric`",
                      "`ts`",
                      "`app`",
                      "`cluster`",
                      "`group`",
                      "`ip`",
                      "`zone`",
                      "`load`",
                      "`memory`",
                      "`cpu`"
                    ],
                    "url": "http://localhost:3306",
                    "schema": "datax_test",
                    "table": "datax_test_multi_field",
                    "writeMode": "insert",
                    "opIndex": "0",
                    "batchSize": "2"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write single-value data to an AnalyticDB for MySQL database. In addition, tags are specified to filter time series.
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "__value__"
                    ],
                    "metric": [
                      "m"
                    ],
                    "tag": {
                      "m": {
                        "app": "a1",
                        "cluster": "c1"
                      }
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"ads",
                  "parameter": {
                    "username": "******",
                    "password": "******",
                    "column": [
                      "`metric`",
                      "`ts`",
                      "`app`",
                      "`cluster`",
                      "`group`",
                      "`ip`",
                      "`zone`",
                      "`value`"
                    ],
                    "url": "http://localhost:3306",
                    "schema": "datax_test",
                    "table": "datax_test",
                    "writeMode": "insert",
                    "opIndex": "0",
                    "batchSize": "2"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write multi-value data to an AnalyticDB for MySQL database. In addition, tags are specified to filter time series.
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "load",
                      "memory",
                      "cpu"
                    ],
                    "metric": [
                      "m_field"
                    ],
                    "field": {
                      "m_field": [
                        "load",
                        "memory",
                        "cpu"
                      ]
                    },
                    "tag": {
                      "m_field": {
                        "ip": "i999"
                      }
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"ads",
                  "parameter": {
                    "username": "******",
                    "password": "******",
                    "column": [
                      "`metric`",
                      "`ts`",
                      "`app`",
                      "`cluster`",
                      "`group`",
                      "`ip`",
                      "`zone`",
                      "`load`",
                      "`memory`",
                      "`cpu`"
                    ],
                    "url": "http://localhost:3306",
                    "schema": "datax_test",
                    "table": "datax_test_multi_field",
                    "writeMode": "insert",
                    "opIndex": "0",
                    "batchSize": "2"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write single-value data to another Alibaba Cloud TSDB database.
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "TSDB",
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "m"
                    ],
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"tsdb",
                  "parameter": {
                    "endpoint": "http://localhost:8240"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In the following code, a node is configured to read data from an Alibaba Cloud TSDB database and write multi-value data to another Alibaba Cloud TSDB database.
    {
        "type":"job",
        "version":"2.0",// The version number.
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "TSDB",
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "m_field"
                    ],
                    "field": {
                      "m_field": [
                        "load",
                        "memory",
                        "cpu"
                      ]
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"tsdb",
                  "parameter": {
                    "multiField": true,
                    "endpoint": "http://localhost:8240"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed.
            },
            "speed":{
                "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
                "concurrent":1,// The maximum number of concurrent threads.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }