TSDB Reader reads data from Time Series Database (TSDB). This topic describes the data types and parameters that are supported by TSDB Reader and how to configure TSDB Reader by using the codeless user interface (UI) and code editor.

Notice TSDB Reader supports only exclusive resource groups for Data Integration, but not the shared resource group or custom resource groups for Data Integration. For more information, see Create and use an exclusive resource group for Data Integration and Create a custom resource group for Data Integration.

Background information

TSDB is a high-performance, cost-effective, stable, and reliable online time series database service. TSDB features high read and write performance and provides a high compression ratio for data storage. TSDB also enables the interpolation and aggregation of time series data. TSDB can monitor devices and business and send alerts in real time. It is widely used in Internet and IoT fields. For more information, see What is TSDB?

TSDB Reader connects to a TSDB instance by sending an HTTP request and obtains data points by using the /api/query or /api/mquery HTTP API endpoint. For more information about HTTP API endpoints, see HTTP API overview. TSDB Reader splits a synchronization node into multiple tasks based on time series and a specific time range.

Limits

  • If the amount of data that is reported for a metric within an hour is greater than a specific threshold, you must modify the value of the -j parameter to change the memory size of your Java Virtual Machine (JVM).
    If the data write speed of the downstream writer is lower than the data read speed of TSDB Reader, a task backlog may occur. Therefore, you must change the memory size of the JVM to an appropriate value. For example, to extract data from an Alibaba Cloud TSDB database to an on-premises data source, you can run the following command in which the JVM memory size is specified:
    python datax/bin/datax.py tsdb2stream.json -j "-Xms4096m -Xmx4096m"
  • The specified start time and end time are automatically converted to on-the-hour time.

    For example, if you set the time range to [3:35, 4:55) of April 18, 2019, the time range is converted to [3:00, 4:00).

Data types

The following table lists the data types that are supported by TSDB Reader.
Data Integration data type TSDB data type
STRING String to which a data point in TSDB is serialized. The data point can be a timestamp, metric, tag, field, and value.

Parameters

Parameter Description Required Default value
name The name of the reader. Yes tsdbreader
sinkDbType The type of the source database.
Valid values: TSDB and RDB.
  • The value TSDB indicates that the source database is an Alibaba Cloud TSDB, OpenTSDB, InfluxDB, Prometheus, or Timescale database.
  • The value RDB indicates that the source database is a relational database, such as an AnalyticDB for MySQL, Oracle, MySQL, PostgreSQL, or Distributed Relational Database Service (DRDS) database.
No TSDB
endpoint The HTTP endpoint of the source TSDB database. Specify the endpoint in the http://IP address:Port number format. Yes No default value
username The username that is used to connect to the source TSDB database. Yes No default value
password The password that is used to connect to the source TSDB database. Yes No default value
column The names of the columns from which you want to read data. The value of this parameter varies based on the value of the sinkDbType parameter.
  • If the value of the sinkDbType parameter is TSDB, set the column parameter to the names of the metrics that you want to read.
  • If the value of the sinkDbType parameter is RDB, set the column parameter to the table fields from which you want to read data. You can also specify the __metric__, __ts__, and __value__ fields in the column parameter based on your business requirements.

    The __metric__ field is used to read metrics. The __ts__ field is used to read timestamps. The __value__ field is used to read single-value data. To read multi-value data, specify the field parameter.

Yes No default value
metric The metrics that you want to read. This parameter is valid only when the sinkDbType parameter is set to RDB. No No default value
field The fields that you want to read. This parameter is valid only for multi-value data synchronization. No No default value
tag The tags that you want to read. Specify the tags in the format of key-value pairs. These tags are used to filter time series. No No default value
splitIntervalMs The interval at which TSDB Reader reads data from the source database. A synchronization node is split into multiple tasks to read data based on the interval. Unit: milliseconds. Yes No default value
beginDateTime The start time of the time range of the data points that you want to read. Specify the start time in the format of yyyy-MM-dd HH:mm:ss. The beginDateTime and endDateTime parameters must be used in pairs.
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) of April 18, 2019, the time range is converted to [3:00, 4:00).
Yes No default value
endDateTime The end time of the time range of the data points that you want to read. Specify the end time in the format of yyyy-MM-dd HH:mm:ss. The beginDateTime and endDateTime parameters must be used in pairs.
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) of April 18, 2019, the time range is converted to [3:00, 4:00).
Yes No default value
combine Specifies whether to combine multiple rows into a single row in the output.

If the source table is a sparse table, you can set this parameter to true for TSDB Reader to combine multiple rows into a single row in the output. However, TSDB Reader combines only rows that have the same timestamp.

If the combine parameter is set to true, TSDB Reader exports values by using __metric__.xxx. In this case, you do not need to specify the __value__ field. To make sure that metrics can be obtained, specify the __metric__.xxx field at the beginning of the array that is specified in the column parameter.

No false

Configure TSDB Reader by using the codeless UI

This method is not supported.

Configure TSDB Reader by using the code editor

For more information about how to configure a synchronization node by using the code editor, see Create a sync node by using the code editor.
  • In this example, a synchronization node is configured to read time series data from an Alibaba Cloud TSDB database and write the data to an on-premises data source.
    The following code provides sample time series data:
    {"metric":"m","tags":{"app":"a19","cluster":"c5","group":"g10","ip":"i999","zone":"z1"},"timestamp":1546272263,"value":1}
    The following code provides the configuration of the synchronization node:
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "TSDB", // The type of the source database.  
                    "endpoint": "http://localhost:8242",
                    "username": "The username that is used to connect to the source database.", 
                    "password": "The password that is used to connect to the source database.",
                    "column": [
                      "m"
                    ],
                    "combine":false,
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"stream",
                "parameter":{
    
                },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads. 
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In this example, a synchronization node is configured to read relational data from an Alibaba Cloud TSDB database and write the data to an on-premises data source.
    The following code provides sample relational data:
    m 1546272125 a1    c1 g2 i3021 z4 1.0
    The following code provides the configuration of the synchronization node:
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB", // The type of the source database.  
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "__value__"
                    ],
                    "metric": [
                      "m"
                    ],
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"stream",
                "parameter":{
    
                },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads. 
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In this example, a synchronization node is configured to read single-value data from an Alibaba Cloud TSDB database and write the data to an AnalyticDB for MySQL database.
    
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "__value__"
                    ],
                    "metric": [
                      "m"
                    ],
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"stream",
                "parameter":{
    
                },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads. 
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In this example, a synchronization node is configured to read multi-value data from an Alibaba Cloud TSDB database and write the data to an AnalyticDB for MySQL database.
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "username": "The username that is used to connect to the source database.",
                    "password": "The password that is used to connect to the source database.",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "load",
                      "memory",
                      "cpu"
                    ],
                    "metric": [
                      "m_field"
                    ],
                    "field": {
                      "m_field": [
                        "load",
                        "memory",
                        "cpu"
                      ]
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"ads",
                  "parameter": {
                    "username": "******",
                    "password": "******",
                    "column": [
                      "`metric`",
                      "`ts`",
                      "`app`",
                      "`cluster`",
                      "`group`",
                      "`ip`",
                      "`zone`",
                      "`load`",
                      "`memory`",
                      "`cpu`"
                    ],
                    "url": "http://localhost:3306",
                    "schema": "datax_test",
                    "table": "datax_test_multi_field",
                    "writeMode": "insert",
                    "opIndex": "0",
                    "batchSize": "2"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads. 
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In this example, a synchronization node is configured to read single-value data in a specific time range from an Alibaba Cloud TSDB database and write the data to an AnalyticDB for MySQL database.
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "username": "The username that is used to connect to the source database.",
                    "password": "The password that is used to connect to the source database.",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "__value__"
                    ],
                    "metric": [
                      "m"
                    ],
                    "tag": {
                      "m": {
                        "app": "a1",
                        "cluster": "c1"
                      }
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"ads",
                  "parameter": {
                    "username": "******",
                    "password": "******",
                    "column": [
                      "`metric`",
                      "`ts`",
                      "`app`",
                      "`cluster`",
                      "`group`",
                      "`ip`",
                      "`zone`",
                      "`value`"
                    ],
                    "url": "http://localhost:3306",
                    "schema": "datax_test",
                    "table": "datax_test",
                    "writeMode": "insert",
                    "opIndex": "0",
                    "batchSize": "2"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads. 
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In this example, a synchronization node is configured to read multi-value data in a specific time range from an Alibaba Cloud TSDB database and write the data to an AnalyticDB for MySQL database.
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "RDB",
                    "endpoint": "http://localhost:8242",
                    "username": "The username that is used to connect to the source database.",
                    "password": "The password that is used to connect to the source database.",
                    "column": [
                      "__metric__",
                      "__ts__",
                      "app",
                      "cluster",
                      "group",
                      "ip",
                      "zone",
                      "load",
                      "memory",
                      "cpu"
                    ],
                    "metric": [
                      "m_field"
                    ],
                    "field": {
                      "m_field": [
                        "load",
                        "memory",
                        "cpu"
                      ]
                    },
                    "tag": {
                      "m_field": {
                        "ip": "i999"
                      }
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"ads",
                  "parameter": {
                    "username": "******",
                    "password": "******",
                    "column": [
                      "`metric`",
                      "`ts`",
                      "`app`",
                      "`cluster`",
                      "`group`",
                      "`ip`",
                      "`zone`",
                      "`load`",
                      "`memory`",
                      "`cpu`"
                    ],
                    "url": "http://localhost:3306",
                    "schema": "datax_test",
                    "table": "datax_test_multi_field",
                    "writeMode": "insert",
                    "opIndex": "0",
                    "batchSize": "2"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads. 
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In this example, a synchronization node is configured to read single-value data from an Alibaba Cloud TSDB database and write the data to another Alibaba Cloud TSDB database.
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "TSDB",
                    "endpoint": "http://localhost:8242",
                    "username": "The username that is used to connect to the source database.",
                    "password": "The password that is used to connect to the source database.",
                    "column": [
                      "m"
                    ],
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"tsdb",
                  "parameter": {
                    "endpoint": "http://localhost:8240"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads. 
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }
  • In this example, a synchronization node is configured to read multi-value data from an Alibaba Cloud TSDB database and write the data to another Alibaba Cloud TSDB database.
    {
        "type":"job",
        "version":"2.0",// The version number. 
        "steps":[
            {
                "stepType":"tsdb",
                  "parameter": {
                    "sinkDbType": "TSDB",
                    "endpoint": "http://localhost:8242",
                    "username": "The username that is used to connect to the source database.",
                    "password": "The password that is used to connect to the source database.",
                    "column": [
                      "m_field"
                    ],
                    "field": {
                      "m_field": [
                        "load",
                        "memory",
                        "cpu"
                      ]
                    },
                    "splitIntervalMs": 60000,
                    "beginDateTime": "2019-01-01 00:00:00",
                    "endDateTime": "2019-01-01 01:00:00"
                  },
                "name":"Reader",
                "category":"reader"
            },
            {
                "stepType":"tsdb",
                  "parameter": {
                    "multiField": true,
                    "endpoint": "http://localhost:8240"
                  },
                "name":"Writer",
                "category":"writer"
            }
        ],
        "setting":{
            "errorLimit":{
                "record":"0"// The maximum number of dirty data records allowed. 
            },
            "speed":{
                "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                "concurrent":1, // The maximum number of parallel threads.  
                "mbps":"12"// The maximum transmission rate.
            }
        },
        "order":{
            "hops":[
                {
                    "from":"Reader",
                    "to":"Writer"
                }
            ]
        }
    }