Integrate aliyun-timestream with Prometheus APIs - Elasticsearch

Alibaba Cloud Elasticsearch provides the aliyun-timestream plug-in for storage and usage enhancement of time series data. The plug-in allows you to execute Prometheus Query Language (PromQL) statements to query data in Elasticsearch. The plug-in can also be seamlessly integrated with Prometheus and Grafana. This topic describes how to use Prometheus APIs based on the aliyun-timestream plug-in.

Background information

aliyun-timestream is a plug-in developed by the Alibaba Cloud Elasticsearch team based on the features of time series products that are provided by the Elastic community. This plug-in is used to enhance the storage and usage performance of time series data. aliyun-timestream uses Prometheus Querying Language (PromQL) statements instead of domain-specific language (DSL) statements to query stored metric data. This helps simplify query operations and improve query efficiency. aliyun-timestream also reduces storage costs. For more information, see Overview of aliyun-timestream.

Prerequisites

An Elasticsearch cluster of the Standard Edition that meets the following version requirements is created: The version of the cluster is V7.16 or later and the kernel version of the cluster is V1.7.0 or later, or the version of the cluster is V7.10 and the kernel version of the cluster is V1.8.0 or later. For information about how to create an Elasticsearch cluster, see Create an Alibaba Cloud Elasticsearch cluster.

Descriptions of Prometheus APIs

You can use Prometheus APIs to query data that is written by using the remote write API of Prometheus or data in any time series index.

If you use a Prometheus API to query data, the prefixes and suffixes of metric fields and dimension fields that are returned are not displayed. If you use a Prometheus API to query data that is written based on the default time series data model, the prefix metrics. of metric fields that are returned are not displayed, and the prefix labels. of dimension fields that are returned are not displayed.

For example, the following data is written by using the remote write API of Prometheus:

{
  "labels": {
    "instance": "127.0.0.1:9114",
    "job": "elasticsearch"
  },
  "metrics": {
    "up": 0.0,
    "scrape_samples_post_metric_relabeling": 0.0,
    "scrape_samples_scraped": 0.0,
    "scrape_duration_seconds": 4.45999E-4,
    "scrape_series_added": 0.0
  },
  "@timestamp": "1655717638795"
}

If you use a Prometheus API to query the preceding data, the prefix labels. of dimension fields that are returned are not displayed. The following code provides an example for the response of using the /api/v1/labels API to query the preceding data:

{
  "status" : "success",
  "data" : [
    "__name__",
    "instance",
    "job"
  ]
}

If you use a Prometheus API to query the preceding data, the prefix metrics. of metric fields that are returned are not displayed. The following code provides an example for the response of using the /api/v1/metadata API to query the preceding data:

{
  "status" : "success",
  "data" : {
    "scrape_samples_post_metric_relabeling" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "scrape_samples_scraped" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "scrape_duration_seconds" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "scrape_series_added" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "up" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ]
  }
}

If you customize settings when you create a time series index, you must specify prefixes and suffixes for the metric fields and dimension fields that you upload. Otherwise, when you use a Prometheus API to query data in the time series index, the actual data is returned. The following code provides an example on how to specify such prefixes and suffixes:

PUT _time_stream/{name}
{
  "time_stream": {
    "labels_fields": "@labels.*_l",
    "metrics_fields": "@metrics.*_m",
    "label_prefix": "@labels.",
    "label_suffix": "_l",
    "metric_prefix": "@metrics.",
    "metric_suffix": "_m"
  }
}

Remote write API

Request syntax

POST /_time_stream/prom_write/{index}
---PB data---

Usage notes

If you configure the remote_write parameter when you install Prometheus, you can use Prometheus to synchronize data to Elasticsearch. The following code provides a configuration example:

remote_write:
  - url: "http://127.0.0.1:9200/_time_stream/prom_write/prom_index"
    basic_auth:
     username: elastic
     password: xxxx

Prometheus uses the remote write API to write data to time series indexes based only on the default time series data model in which the prefix of dimension fields is labels. and the prefix of metric fields is metrics..

Instant query API

Request syntaxes

GET  /_time_stream/prom/{index}/query
POST /_time_stream/prom/{index}/query

Usage notes

For information about the instance query API of Prometheus, see Prometheus instant queries.

Range query API

Request syntaxes

GET  /_time_stream/prom/{index}/query_range
POST /_time_stream/prom/{index}/query_range

Usage notes

For information about the range query API of Prometheus, see Prometheus range queries.

Series metadata API

Request syntaxes

GET  /_time_stream/prom/{index}/series
POST /_time_stream/prom/{index}/series

Usage notes

The series metadata API of Prometheus is used to query the list of time series. For more information about this API, see Prometheus finding series by label matchers.

Label metadata API

Request syntaxes

GET  /_time_stream/prom/{index}/labels
POST /_time_stream/prom/{index}/labels

Usage notes

The label metadata API of Prometheus is used to query the list of dimensions. For more information about this API, see Prometheus getting label names.

Label value metadata API

Request syntax

GET /_time_stream/prom/{index}/label/<label_name>/values

Usage notes

The label value metadata API of Prometheus is used to query the list of dimension values. For more information about this API, see Prometheus querying label values.

Metric metadata API

Request syntax

GET /_time_stream/prom/{index}/metadata

Usage notes

The metric metadata API of Prometheus is used to query the list of metrics. For more information about this API, see Prometheus querying metric metadata.

Support of aliyun-timestream for PromQL

Support for expressions

Expression	Type	Supported or not	Description
powOp	binaryOperator	Yes	Power
unaryOp	binaryOperator	Yes	Positive and negative
multOp	binaryOperator	Yes	Multiplication, division, and modulo
addOp	binaryOperator	Yes	Addition and subtraction
compareOp	binaryOperator	Yes	Comparison (==, >=, >, <=, <, and !=)
andUnlessOp	binaryOperator	Yes	Set operation (and, or, and unless)
orOp	binaryOperator	Yes	Set operation (or)
function	vector	Yes	Function
aggregation	vector	Yes	Aggregation operation
instantSelector	vector	Yes	Instant selector
matrixSelector	vector	Yes	Vector selector
offset	vector	Yes	Offset
literal	vector	Yes	Constant
labelMatcher	vector	Yes	Label matching
grouping	group	No	Grouping aggregation, including on, ignoring, group_left, and group_right

Support for aggregation operators

Aggregation operator	Supported or not
sum(calculate sum over dimensions)	Yes
min(select minimum over dimensions)	Yes
max(select maximum over dimensions)	Yes
avg(calculate the average over dimensions)	Yes
group(all values in the resulting vector are 1)	No
stddev(calculate population standard deviation over dimensions)	No
stdvar(calculate population standard variance over dimensions)	No
count(count number of elements in the vector)	Yes
count_values(count number of elements with the same value)	Yes
bottomk(smallest k elements by sample value)	Yes
topk(largest k elements by sample value)	Yes
quantile(calculate φ-quantile (0 ≤ φ ≤ 1) over dimensions)	Yes

Support for functions

Function	Supported or not
abs(v instant-vector)	Yes
absent(v instant-vector)	No
absent_over_time(v range-vector)	No
ceil(v instant-vector)	Yes
changes(v range-vector)	No
clamp(v instant-vector, min scalar, max scalar)	Yes
clamp_max(v instant-vector, max scalar)	Yes
clamp_min(v instant-vector, min scalar)	Yes
day_of_month(v=vector(time()) instant-vector)	Yes
day_of_week(v=vector(time()) instant-vector)	Yes
days_in_month(v=vector(time()) instant-vector)	No
delta(v range-vector)	Yes
deriv(v range-vector)	No
exp(v instant-vector)	Yes
floor(v instant-vector)	Yes
histogram_quantile(φ scalar, b instant-vector)	No
holt_winters(v range-vector, sf scalar, tf scalar)	No
hour(v=vector(time()) instant-vector)	Yes
idelta(v range-vector)	Yes
increase(v range-vector)	Yes
irate(v range-vector)	Yes
label_join(v instant-vector, dst_label string, separator string, src_label_1 string, src_label_2 string, ...)	No
label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string)	No
ln(v instant-vector)	Yes
log2(v instant-vector)	No
log10(v instant-vector)	Yes
minute(v=vector(time()) instant-vector)	Yes
month(v=vector(time()) instant-vector)	Yes
predict_linear(v range-vector, t scalar)	No
rate(v range-vector)	Yes
resets(v range-vector)	No
round(v instant-vector, to_nearest=1 scalar)	No
scalar(v instant-vector)	No
sgn(v instant-vector)	Yes
sort(v instant-vector)	No
sort_desc()	No
sqrt(v instant-vector)	Yes
time()	No
timestamp(v instant-vector)	Yes
vector(s scalar)	No
year(v=vector(time()) instant-vector)	Yes
avg_over_time(range-vector)	Yes
min_over_time(range-vector)	Yes
max_over_time(range-vector)	Yes
sum_over_time(range-vector)	Yes
count_over_time(range-vector)	Yes
quantile_over_time(scalar, range-vector)	Yes
stddev_over_time(range-vector)	No
stdvar_over_time(range-vector)	No
last_over_time(range-vector)	Yes
present_over_time(range-vector)	No
acos(v instant-vector)	Yes
acosh(v instant-vector)	Yes
asin(v instant-vector)	Yes
asinh(v instant-vector)	No
atan(v instant-vector)	Yes
atanh(v instant-vector)	No
cos(v instant-vector)	Yes
cosh(v instant-vector)	Yes
sin(v instant-vector)	Yes
sinh(v instant-vector)	Yes
tan(v instant-vector)	Yes
tanh(v instant-vector)	Yes
deg(v instant-vector)	Yes
pi()	Yes
rad(v instant-vector)	Yes