OpenTSDB is a scalable and distributed time series database that is maintained by Yahoo. OpenTSDB Reader allows you to read data from OpenTSDB.

You can configure OpenTSDB Reader only by using the code editor.

How it works

OpenTSDB Reader connects to an OpenTSDB instance by sending an HTTP request, and obtains the connection information of the underlying HBase data store by using the /api/config HTTP API operation. Then, OpenTSDB Reader connects to HBase by using the AsyncHBase client and reads data points by scanning the data points. A sync node is split to multiple export tasks based on the database, metric, and time range. An export task exports the data of the specified metric in the specified hour.

Limits

  • The specified start time and end time are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
  • Only OpenTSDB 2.3.x is supported.
  • You cannot directly query the data points by using the /api/query HTTP API operation. Before you query data points, connect to the underlying data store of OpenTSDB.

    If you directly use the /api/query HTTP API operation to read a large amount of data points, the AsyncHBase client of OpenTSDB reports an exception of excessive callbacks. To avoid this issue, OpenTSDB Reader connects to the underlying HBase data store and reads data points by scanning the data points. To improve query efficiency, you can specify the metric and time range to allow OpenTSDB Reader to scan HBase tables in sequence.

Data types

Category Data Integration data type OpenTSDB data type
String string String to which a data point in OpenTSDB is serialized, including the timestamp, metric, tags, and value

Parameters

Parameter Description Required Default value
endpoint The HTTP endpoint for connecting to OpenTSDB, in the format of http://IP:Port. Yes N/A
column The metrics to be synchronized. Yes N/A
beginDateTime The start time of the time range of the data points to be migrated, in the format of yyyyMMddHHmmss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be migrated. Yes N/A
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
endDateTime The end time of the time range of the data points to be migrated, in the format of yyyyMMddHHmmss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be migrated. Yes N/A
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).

Configure OpenTSDB Reader by using the codeless UI

The codeless user interface (UI) is not supported for OpenTSDB Reader.

Configure OpenTSDB Reader by using the code editor

The following example shows how to configure a sync node to read data from an OpenTSDB database. For more information, see Create a sync node by using the code editor.
{
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "concurrent": 1,
            "throttle": true
        }
    },
    "steps": [
        {
            "category": "reader",
            "name": "Reader",
            "parameter": {
                "endpoint": "http://host:4242",
                "column": [
                    "xc"
                ],
                "beginDateTime": "20190101000000",
                "endDateTime": "20190101030000"
            },
            "stepType": "opentsdb"
        },
        {
            "category": "writer",
            "name": "Writer",
            "parameter": {},
            "stepType": ""
        }
    ],
    "type": "job",
    "version": "2.0"
}

Performance test report

  • Characteristics of test data
    The data that is used in the performance test is described by using the following parameters:
    • Metric: a metric, which is m.
    • tag_k and tag_v: the key and value of the tag. The keys and values of the first four tags constitute a time series of 2,000,000 data points, that is, 10 (zones) × 20 (clusters) × 100 (groups) × 100 (apps). The ip tag corresponds to the index of the 2,000,000 data points, starting from 1.
      tag_k tag_v
      zone z1~z10
      cluster c1~c20
      group g1~100
      app a1~a100
      ip ip1~ip2,000,000
    • value: a random value from 1 to 100.
    • interval: a collection period of 10 seconds. The data collection lasts for 3 hours, and a total of 2,160,000,000 data points, namely, 3 × 60 × 60/10 × 2,000,000, are collected.
  • Performance test results
    Number of channels Data integration speed (records per second) Data integration bandwidth (Mbit/s)
    1 215,428 25.65
    2 424,994 50.60
    3 603,132 71.81