FAQ about metric storage - Simple Log Service - Alibaba Cloud Documentation Center

Why do the results of the `rate` and `increase` functions sometimes yield abnormally large values?

These functions are applicable only to incrementing metric data, where values increase over time. The anomaly can be explained by the Prometheus Engine source code.

The screenshot below shows how the rate and increase functions calculate results over a time window: first, the difference between the first two values is assigned to resultFloat. If subsequent values decrease, the previous value (prevValue) is added to resultFloat, leading to an abnormally large final value.

Troubleshooting

Use PromQL to check for any value decreases. First, narrow the time filter to the period when the anomaly occurred, and execute the following PromQL query (replace xxxx_metric with the actual metric name). If the result set contains data points, this indicates a "value decrease".
```
Query: (xxxx_metric{} - xxxx_metric{} offset 1s) < 0
Step: 1s
```

For the time frame with abnormal values, retrieve the original data points using SQL, such as:

* | select *, from_unixtime(__time_nano__/1000000.0)  from "metricstore_name.prom" where __name__='metric_name' and element_at(__labels__, 'filter_labelKey')='filter_labelValue' order by __time_nano__

Note

We recommend that you use multiple element_at functions in SQL to further narrow the timeframe.

The SQL result set is sorted by timestamp, allowing for easy observation of the raw erroneous data points.

What do I do if no data is queried by executing a PromQL statement when metrics were already written to a specific Metricstore?

Check whether the corresponding issues exist in the following scenarios:

Scenario 1: The syntax parsing status is automatically displayed in the search box of the Metricstore to indicate whether the syntax of the specified PromQL statement is valid. If the syntax is invalid, modify it as prompted.

Scenario 2: Go to the Custom Analysis page of the Metricstore, as shown in the following figure. In the search box, execute the following SQL statement to check whether data exists within the specified period of time.
The __name__ field in the SQL statement specifies the metric name.
```
* | select * from "<Metricstore name>.prom" where __name__ = 'demo_api_request_duration_seconds_bucket'
```

If no issues exit in the preceding two scenarios, this error occurs due to the vector selection logic and lookback-delta mechanism of the Prometheus Query compute engine.

The Prometheus Query compute engine provides a query language named PromQL. When you use PromQL to query data, not all data points are computed. Before each query operation is performed, a vector selection operation is performed. All operators and functions that are supported in the PromQL syntax can be classified into the following categories based on the vector selection process: range vector selectors that end with a time duration in brackets, such as [1m] or [1h], and instant vector selectors. For more information, see Range Vector Selectors and Instant vector selectors.

The following sample code provides examples on how to use vector selectors:

RangeVectorSelector:
rate(http_requests_total[5m])
delta(http_requests_total[5m])
count_over_time(http_requests_total[5m])


InstantVector:
http_requests_total
absent(http_requests_total)
count(http_requests_total)

A range vector selector selects a range of samples by computing all data points within the time range specified by the [xx] operator, as shown in the following figure. The operator count_over_time ( up [30s] ) indicates that all data points within the last 30 seconds are computed.

The lookback-delta mechanism in PromQL is valid only for instant vector selectors. An instant vector selector backtracks the data within a specific time interval based on the value of the lookback-delta parameter, and uses the data point at the nearest point in time as the value for the current point in time. In most cases, the write time of raw data and the query time are not aligned. If you want to query the data value at a specific point in time, the data points within n minutes before the selected point in time are backtracked, and the data point at the nearest point in time is used as the value for the current point in time. By default, Simple Log Service backtracks the data points within 3 minutes before the selected point in time. For information about how to modify the related parameters, see API operations for metric queries.

In this example, the startTime of a query is 00:09:28. If no raw data point exists at this point in time, the system backtracks the data within 3 minutes before 00:09:28. If no data is written from 00:06:28 to 00:09:28, no data point is selected for 00:09:28. Similarly, the data point at 00:09:40 is selected for the query time 00:09:43, and the data point at 00:09:55 is selected for the query time 00:09:58.

However, the following special scenario exists based on the mechanism: Data is queried within a specific period of time by using a PromQL statement even if no data points exist within the period of time. In this example, no data exists after 00:10:00. If you query data at 00:10:13, 00:10:28, 00:10:43, and 00:12:58, the system backtracks the data within 3 minutes before the query time based on the lookback-delta mechanism, and returns the data point at 00:10:00 for the preceding queries.

Instant vector selectors and range vector selectors use different vector selection logic. If no data is queried, troubleshoot the error based on the following scenarios:

RangeVectorSelector
The time interval specified by the operator [xx] is small, and the step size specified by the step parameter is large. In this case, a data point may fail to be selected by the selector before each compute operation.
For example, the raw data of the up metric is written every hour on the hour. Configure the following query parameters:
```
startTime: 10:30:00
endTime  : 18:30:00
step     : 1h
query    : count_over_time(up[10m])
```
In this example, a response with seven data points is expected, but the actual query result is empty. Cause: A compute operation is performed once per step from the startTime. Data is computed at 11:30:00, 12:30:00, 13:30:00, 14:30:00, 15:30:00, 16:30:00, 17:30:00, and 18:30:00. Before each compute operation, the selector backtracks the data within 10 minutes before each query time. However, no data points exist in the specified time interval.

InstantVectorSelector
The value of the lookback-delta parameter is small, and the step size specified by the step parameter is large. In this case, the nearest data point may fail to be selected by the selector before each compute operation.
For example, the raw data of the up metric is written every hour on the hour. Configure the following query parameters:
```
startTime: 10:30:00
endTime  : 18:30:00
step     : 1h
query    : count(up)
```
In Prometheus, the default value of the lookback-delta parameter is 5 minutes. In Simple Log Service Metricstores, the default value is 3 minutes. In this example, data is computed at 11:30:00, 12:30:00, 13:30:00, 14:30:00, 15:30:00, 16:30:00, 17:30:00, and 18:30:00. Before each compute operation, the selector backtracks the data within 3 minutes before each query time and selects the data point at the nearest point in time as the value for the specified query time. Due to the sparse distribution of raw data points, no data point are found in the specified time intervals. As a result, the query result is also empty.

Solution

Modify the startTime, endTime, and step parameters in the query parameters and align the parameter settings with the data write time. Then, data points at the specified data write time can be selected.
In this example, seven data points are returned for the query. In actual scenarios, the write time of metric points is irregular. As a result, the query parameter settings cannot always be aligned with the write time. We recommend that you do not adopt this solution.
```
startTime: 10:00:00
endTime  : 18:00:00
step     : 1h
query    : count(up)
```
Specify a smaller value for the step parameter. For example, set the step parameter to 3 minutes in the preceding InstantVectorSelector scenario. Then, the selection operations are performed at a higher frequency. This solution increases the number of selection operations to ensure that data points can be selected even if the raw data points are sparsely distributed.
Specify a larger value for the lookback-delta parameter or the [xx] operator. This way, the time interval for selections can be large enough to include more write time of raw data points.

What do I do if data is still queried after a specific point in time by using a PromQL statement even if no data is written after the point in time?

This error occurs due to the lookback-delta mechanism of Prometheus.

Why are the timestamps of the data points in the query results inconsistent with the write times?

This error occurs due to the lookback-delta mechanism of Prometheus.

Why do the latest results differ from historical results for the same time range?

There are two possibilities:

If the time filter is set to "relative time," changes to the start and end parameters may lead to different data points being selected during the "lookback-delta" operation, resulting in inconsistent outcomes. This behavior is standard in Prometheus and is considered normal.
If the two executions are spaced out significantly and share the same time range, it may be that not all expected data points had arrived on the Simple Logs Service side during the historical query (due to write delays). Differences in the data used in the calculations can lead to inconsistent results. Use the offset syntax in PromQL to shift the query window back, correcting for errors caused by write delays.

What do I do if my query request is rejected because the consumed compute resources exceed the upper limit per query?

If a query response contains an error message, such as "too many time Series or items, xxxxxx" or "too many time Series or items in parallelMaster node, xxxxx", the query reads a large amount of data and has triggered the memory limit at the computing layer.

Solution

Specify a smaller query time range.
If the execution time heavily relies on the specified time frame, it is advisable to enable concurrent computing.

What do I do if the "vector cannot contain metrics with the same labelset" error message appears?

Cause 1: The label names in the value of the __labels__ field are not sorted in alphabetical order.
The value of the __labels__ field that is written to a Metricstore consists of multiple labels. Each label contains a label name and a label value in the <Label name>#$#<Label value> format. Multiple labels are separated by vertical bars (|). The labels in a metric identifier must be sorted in alphabetical order by label name. For more information, see Metric identifier.
If the label names in the value of the __labels__ field are not sorted in alphabetical order, this error occurs. Execute the following SQL statement to check whether the label names in the value of the __labels__ field are sorted in alphabetical order:
```
* | select * from (
select __labels__, array_join(array_sort(split(__labels__, '|')), '|') as rightLabels from "<Metricstore name>.prom" where __name__!=''
) where __labels__ != rightLabels
```
Cause 2: Empty label values exist in the value of the __labels__ field.
The Prometheus engine considers labels with empty values to be invalid. The Prometheus engine deletes invalid labels during computing. Perform the following steps to check whether empty label values exist in the value of the __labels__ field:
1. Specify a smaller query time range to find the approximate time interval in which this error occurs.
2. Execute the following SQL statement:
```
* | select __labels__ from "<Metricstore name>.prom" where __name__!='' and regexp_like(__labels__, '.*#\$#\|.*|.*#\$#$')
```
  If a result is returned, empty label values exist. We recommend that you modify the code used to implement data reporting to delete invalid labels.

What do I do if the label information that exists in raw data cannot be queried by executing PromQL statements?

If the label names in the value of the __labels__ field are not sorted in alphabetical order, this error occurs. Execute the following SQL statement to check whether the label names in the value of the __labels__ field are sorted in alphabetical order:

* | select * from (
select __labels__, array_join(array_sort(split(__labels__, '|')), '|') as rightLabels from "<Metricstore name>.prom" where __name__!=''
) where __labels__ != rightLabels

What do I do if an unexpected result is returned by a PromQL statement that includes the by or without clause?

* | select * from (
select __labels__, array_join(array_sort(split(__labels__, '|')), '|') as rightLabels from "<Metricstore name>.prom" where __name__!=''
) where __labels__ != rightLabels

What do I do if no data is available for metric exploration?

A possible cause is that no data exists within the default query time range.

Make sure that data exists within the specified time range.

Note

To increase the exploration response speed, the query time range for exploration is the last 5 minutes by default.

What do I do if a large number of time series are missing in the queried metric data?

When you perform query operations on the query and analysis page of a Metricstore in the Simple Log Service console, the limit parameter is used to control the amount of data that can be returned by an SQL or PromQL statement. We recommend that you increase the value of this parameter.

What do I do if the Warning keyword exists in PromQL calculation results?

A possible cause is that the data within the specified query time range failed to be read because the maximum shard read capacity is reached during data pull. We recommend that you specify a smaller query time range or split shards to improve the overall throughput.

What do I do if the "exceeded maximum resolution of 11,000 points per timeseries. Try decreasing the query resolution (?step=XX)" error message appears?

In Prometheus, each time series can contain up to 11,000 samples. The maximum value of (endTime - startTime)/step is 11000. We recommend that you specify a smaller query time range or specify a larger value for the step parameter.

Why do the results of the rate and increase functions sometimes yield abnormally large values?