Prometheus is a time series database developed and maintained by SoundCloud. It is the open-source implementation of BorgMon, which is the monitoring system of Google. Prometheus Reader allows you to read data from Prometheus.

Currently, you can only configure Prometheus Reader by using the code editor.

How it works

Prometheus Reader connects to a Prometheus instance by sending an HTTP request, and obtains raw data points through the /api/v1/query_range HTTP API operation. A sync node is split to multiple export tasks based on the metric and time range.

Limits

  • The specified start time and end time are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
  • Currently, only Prometheus 2.9.x is supported.
  • The default time range for reading data is 10s.

    The /api/v1/query_range HTTP API operation only allows you to query a limited number of data points. If you specify a too large time range, the following error message is returned: exceeded maximum resolution of 11,000 points per timeseries. Therefore, Prometheus Reader obtains data at the time range of 10s by default. Even if the raw data points are stored by millisecond, up to 10,000 data points can be queried through the/api/v1/query_range HTTP API operation.

Data types

Category Data Integration data type Prometheus data type
String String String to which a data point in Prometheus is serialized, including the timestamp, metric, tags, and value

Parameters

Parameter Description Required Default value
endpoint The HTTP endpoint for connecting to Prometheus, in the format of http://IP:Port. Yes None
column The metrics to be migrated. Yes None
beginDateTime The start time of the time range of the data points to be migrated, in the format of yyyyMMddHHmmss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be migrated. Yes None
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).
endDateTime The end time of the time range of the data points to be migrated, in the format of yyyyMMddHHmmss. Specify the beginDateTime and endDateTime parameters to determine the time range of the data points to be migrated. Yes None
Note The start time and end time of the time range are automatically converted to on-the-hour time. For example, if you set the time range to [3:35, 4:55) on April 18, 2019, the time range is converted to [3:00, 4:00).

Configure Prometheus Reader by using the codeless UI

Currently, the codeless user interface (UI) is not supported for Prometheus Reader.

Configure Prometheus Reader by using the code editor

In the following code, a node is configured to read data from a Prometheus database.

```json
{
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "concurrent": 1,
            "throttle": true
        }
    },
    "steps": [
        {
            "category": "reader",
            "name": "Reader",
            "parameter": {
                "endpoint": "http://localhost:9090",
                "column": [
                    "up"
                ],
                "beginDateTime": "20190520150000",
                "endDateTime": "20190520160000"
            },
            "stepType": "prometheus"
        },
        {
            "category": "writer",
            "name": "Writer",
            "parameter": {},
            "stepType": ""
        }
    ],
    "type": "job",
    "version": "2.0"
}
```

Performance test report

Number of channels Data integration speed (records per second) Data integration bandwidth (Mbit/s)
1 45,000 5.36
2 55,384 6.60
3 60,000 7.15