OpenTSDB is a scalable and distributed time series database maintained by Yahoo. OpenTSDB Reader allows you to read data from OpenTSDB.

Currently, you can only configure OpenTSDB Reader by using the code editor.

How it works

OpenTSDB Reader connects to an OpenTSDB instance by sending an HTTP request, and obtains the connection information of the underlying HBase data store through the /api/config HTTP API operation. Then, OpenTSDB Reader connects to HBase through the AsyncHBase client and reads data points through scan operations. A sync node is split to multiple export tasks based on the database, metric, and time range. An export task exports the data of the specified metric in the specified hour.

Limits

  • The specified start time and end time are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
  • Currently, only OpenTSDB 2.3.x is supported.
  • You cannot directly query the data points by using the /api/query HTTP API operation. Before querying data points, connect to the underlying data store of OpenTSDB.

    If you directly use the /api/query HTTP API operation to read a large amount of data points, the AsyncHBase client of OpenTSDB reports an exception of excessive callbacks. To avoid this issue, OpenTSDB Reader connects to the underlying HBase data store and reads data points through scan operations. To improve query efficiency, you can specify the metric and time range to allow OpenTSDB Reader to scan HBase tables in sequence.

Data types

Category Data Integration data type OpenTSDB data type
String String String to which a data point in OpenTSDB is serialized, including the timestamp, metric, tags, and value

Parameters

Parameter Description Required Default value
endpoint The HTTP endpoint for connecting to OpenTSDB, in the format of http://IP:Port. Yes None
column The metrics to be migrated. Yes None
beginDateTime The start time of the time range of the data points to be migrated, in the format of yyyyMMddHHmmss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be migrated. Yes None
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
endDateTime The end time of the time range of the data points to be migrated, in the format of yyyyMMddHHmmss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be migrated. Yes None
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).

Configure OpenTSDB Reader by using the codeless UI

Currently, the codeless user interface (UI) is not supported for OpenTSDB Reader.

Configure OpenTSDB Reader by using the code editor

In the following code, a node is configured to read data from an OpenTSDB database.

```json
{
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "concurrent": 1,
            "throttle": true
        }
    },
    "steps": [
        {
            "category": "reader",
            "name": "Reader",
            "parameter": {
                "endpoint": "http://host:4242",
                "column": [
                    "xc"
                ],
                "beginDateTime": "20190101000000",
                "endDateTime": "20190101030000"
            },
            "stepType": "opentsdb"
        },
        {
            "category": "writer",
            "name": "Writer",
            "parameter": {},
            "stepType": ""
        }
    ],
    "type": "job",
    "version": "2.0"
}
```

Performance test report

  • Characteristics of test data

    The data used in the performance test is described as follows:

    • Metric: a metric, which is m.
    • tag_k and tag_v: the key and value of the tag. The keys and values of the first four tags constitute a time series of 2,000,000 data points, that is, 10 (zones) × 20 (clusters) × 100 (groups) × 100 (apps). The ip tag corresponds to the index of the 2,000,000 data points, starting from 1.
      tag_k tag_v
      zone z1 to z10
      cluster c1 to c20
      group g1 to g100
      app a1 to a100
      ip ip1 to ip2,000,000
    • value: a random value from 1 to 100.
    • interval: a collection period of 10 seconds. The data collection lasts for 3 hours, and a total of 2,160,000,000 data points, that is 3 × 60 × 60/10 × 2,000,000, are collected.
  • Performance test results
    Number of channels Data integration speed (records per second) Data integration bandwidth (Mbit/s)
    1 215,428 25.65
    2 424,994 50.60
    3 603,132 71.81