Query syntax

Last Updated: Oct 01, 2017

Log Service provides a set of query syntaxes used to express query conditions to help you easily query logs. You can specify query conditions through the GetLogs and GetHistograms interfaces in the Log Service API or on the query page of the Log Service console. This section details query condition syntax.

Full text index and key/value index

You can create an index for the LogStore in two modes:

  • Full text index: The entire line of log is queried as a whole, without differentiating the key and value (Key, Value).
  • Key/value index: Query is performed when Key is specified, for example, FILE:app, Type:action. All the contained strings under this key will be hit.

Syntax keyword

Log Service query conditions support the following keywords:

Name Meaning
and Binary operator, in the format of query1 and query2, indicating the intersection of the query results of query1 and query2. If there is no syntax keyword between the words, the relation between the words is and by default.
or Binary operator, in the format of query1 or query2, indicating the union set of the query results of query1 and query2.
not Binary operator, in the format of query1 not query2, indicating a result that meets query1 and does not meet query2, that is, query1–query2. If only not query1 exists, it indicates that logs not containing the query results of query1 are selected.
(,) Left and right brackets are used to merge one or multiple sub-queries into one query to improve the query priority in the brackets.
: Used to query the key-value pair. term1:term2 forms a key-value pair. If the key or value contains spaces, quotation marks need to be used to include the entire key or value.
Converts a keyword into a common query character. Any term in the left and right quotation marks will be queried and will not be used as a syntax keyword. Or all the terms in the left and right quotation marks are regarded as a whole in the key-value query.
\ Escape character. Used to escape quotation marks; the quotation marks after escaping indicate the symbols themselves and will not be considered as escape characters, for example, "\"".
| Pipeline operator, indicating more computing based on the previous computing, for example, query1 | timeslice 1h | count.
timeslice Time slice operator indicates the length of time during which the data is regarded as a whole for computing.The methods of use are timeslice 1h, timeslice 1m, and timeslice 1s, which respectively indicate 1 hour, 1 minute, and 1 second as a whole. For example, query1 | timeslice 1h | count indicates querying the query condition, and the total times with 1 hour as the time spice are returned.
count Count operator, indicating the number of logs.
* Fuzzy query keyword, used to replace 0 or multiple characters. For example: que*; all the hit words starting with que will be returned.
? Fuzzy query keyword, used to replace one character. For example, qu?ry; all the hit words starting with beginning qu, ending with ry, and with a character in the middle.
__topic__ Query the data under a certain topic. Under the new syntax, the data of 0 or more topics can be queried in the query, for example, __topic__:mytopicname.
__tag__ Query a tag value under a tag key, for example, __tag__:tagkey:tagvalue.
source Query the data of an IP, for example, source:127.0.0.1.
source Query the data of an IP, for example, source:127.0.0.1.
> Query the logs with the value of a field greater than a specific number, for example, latency > 100.
>= Query the logs with the value of a field greater than or equal to a specific number, for example, latency >= 100.
< Query the logs with the value of a field less than a specific number, for example, latency < 100.
<= Query the logs with the value of a field less than or equal to a specific number, for example, latency <= 100.
= Query the logs with the value of a field equal to a specific number, for example, latency = 100.
in Query the logs with a field falling within a specific range. Brackets ([]) are used to indicate closed intervals and parentheses (()) indicate open intervals, with two numbers enclosed and separated by spaces. For example, latency in [100 200] or latency in (100 200] .

Note:

  • Syntax keywords are case-insensitive.
  • Priorities of syntax keywords are sorted in descending order as follows : > " > ( ) > and not > or.
  • Log Service reserves the right to use the following keywords: sort asc desc group by avg sum min max limit. If you need to use the following keywords, use quotation marks to contain them.
  • If the full text index and the key/value index have different word segmentation characters when they are configured, data cannot be queried using the full text query method.
  • To perform a numeric query, the data type of the queried column must be set as double or long. If no data type is set or the syntax used for the numeric range query is incorrect, Log Service translates the query condition as a full text index, which can lead to an unexpected result.
  • If you change the data type of a column from text to numeric, only the = query is supported for the data prior to this change.

Query examples

  1. Logs that contains a and b at the same time: a and b or a b.
  2. Logs that contain a or b: a or b.
  3. Logs that contain a but no b: a not b.
  4. Those in all the logs that contain no a: not a.
  5. Query the logs that contain a and b, but no c: a and b not c.
  6. Logs that contain a or b and must contain c: (a or b ) and c.
  7. Logs that contain a or b, but no c: (a or b ) not c.
  8. Logs that contain a and b and may contain c: a and b or c.
  9. Logs whose FILE field contains apsara: FILE:apsara.
  10. Logs whose FILE field contains apsara and shennong: FILE:"apsara shennong" or FILE:apsara FILE: shennong or FILE:apsara and FILE:shennong.
  11. Logs containing and: and.
  12. Logs with the FILE field containing apsara or shennong: FILE:apsara or FILE:shennong.
  13. Logs with the file info field containing apsara: "file info":apsara.
  14. Logs that contain quotation marks: \".
  15. Logs starting with shen: shen*.
  16. Query all the logs starting with shen under the FILE field: FILE:shen*.
  17. Query the logs starting with shen, ending with ong end and with a character in the middle: shen?ong.
  18. Query all the logs starting with shen and aps: shen* and aps*.
  19. Query the distribution of logs starting with shen, with the time slice as 20 minutes: shen*| timeslice 20m | count.
  20. Query all the data under topic1 and topic2: __topic__:topic1 or __topic__ : topic2.
  21. Query all the data of tagvalue2 under tagkey1: __tag__ : tagkey1 : tagvalue2.
  22. Query for all the data with a latency greater than or equal to 100 and less than 200 can be written in either of the following ways: latency >=100 and latency < 200 or latency in [100 200).
  23. Query for all the requests with a latency greater than 100 must be written in the following way: latency > 100.

Additional query types

Specified or cross-topic query

Each LogStore can be divided into one or more subspaces according to the topic. During query, the query range can be limited for the specified topic to increase the speed. This means that users with a level-2 classification requirement for the LogStore are recommended to use topic to divide the LogStore.

When one or more topics are specified to perform query, query is only implemented in the topic that meets the conditions. However, if no topic is entered, the data under all the topics is queried by default.

Example in which topics are used to classify logs under different domain names:

topic

Topic query example:

  • The data under all the topics can be queried. The data of all the topics will be queried if no topic is specified in the query syntax and parameter.
  • Topic can be queried in the query. The query syntax is __topic__:topicName. The old mode is still supported at the same time. The topic is specified in the URL parameter.
  • Multiple topics can be queried, for example, __topic__:topic1 or __topic__:topic2 indicates the union of data under topic1 and topic2.

Histogram query

The added syntax of Log Service provides the custom interval function. The query syntax is as follows:

  1. where_condition | timeslice 1[hms] |count
  2. h stands for the unit of hour
  3. m stands for the unit of minute
  4. s stands for the unit of second

The interval size can be adjusted by changing the timeslice parameter. For example, to query the data of 2 hours, the corresponding results of different timeslice parameters are as follows:

timeslice parameter Number of Histogram intervals Size of each interval
1h 2 1 hour
30m 4 30 minutes
2m 60 2 minutes
30s 240 30 seconds
Thank you! We've received your feedback.