Prometheus is a time series database (TSDB) that is developed and maintained by SoundCloud. Prometheus is the open source implementation of BorgMon, the monitoring system of Google. Prometheus Reader can read data from Prometheus databases.

Notice

How it works

Prometheus Reader connects to a Prometheus instance by sending an HTTP request and obtains raw data points by using the HTTP API endpoint /api/v1/query_range. A synchronization node is split into multiple tasks based on the metric and time range.

Limits

  • The specified start time and end time are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) of April 18, 2019, the time range is converted to [3:00, 4:00).
  • Prometheus Reader supports only Prometheus 2.9.x.
  • The time range that you specify is divided at a granularity of 10 seconds by default.

    The /api/v1/query_range endpoint allows you to query only a limited number of data points. If you specify an excessively large time range, the following error message is returned: exceeded maximum resolution of 11,000 points per timeseries. Therefore, Prometheus Reader obtains data at a granularity of 10 seconds by default. Even if the raw data points are stored by millisecond, you can query only a maximum of 10,000 data points by using the/api/v1/query_range endpoint.

Data types

The following table lists the mapping between the Data Integration data type and Prometheus data type.

Category Data Integration data type Prometheus data type
String STRING String to which a data point in Prometheus is serialized. The data point can be a timestamp, metric, tag, or value.

Parameters

Parameter Description Required Default value
endpoint The HTTP endpoint of Prometheus, in the format of http://IP address:Port number. Yes No default value
column The metrics from which you want to read data points. Yes No default value
beginDateTime The start time of the time range of the data points that you want to read, in the format of yyyyMMddHHmmss. The beginDateTime parameter and the endDateTime parameter must be used in pairs. Yes No default value
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) of April 18, 2019, the time range is converted to [3:00, 4:00).
endDateTime The end time of the time range of the data points that you want to read, in the format of yyyyMMddHHmmss. The endDateTime parameter and the beginDateTime parameter must be used in pairs. Yes No default value
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) of April 18, 2019, the time range is converted to [3:00, 4:00).

Configure Prometheus Reader by using the codeless UI

This method is not supported.

Configure Prometheus Reader by using the code editor

In the following code, a synchronization node is configured to read data from a Prometheus database. For more information about how to configure a synchronization node by using the code editor, see Create a sync node by using the code editor. 
{
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "concurrent": 1,// The maximum number of parallel threads.
            "throttle": true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "steps": [
        {
            "category": "reader",
            "name": "Reader",
            "parameter": {
                "endpoint": "http://localhost:9090",
                "column": [
                    "up"
                ],
                "beginDateTime": "20190520150000",
                "endDateTime": "20190520160000"
            },
            "stepType": "prometheus"
        },
        {
            "category": "writer",
            "name": "Writer",
            "parameter": {},
            "stepType": ""
        }
    ],
    "type": "job",
    "version": "2.0"
}

Performance test report

Number of channels Data integration speed (records/s) Data integration bandwidth (Mbit/s)
1 45,000 5.36
2 55,384 6.60
3 60,000 7.15