Query syntax

Last Updated: Dec 29, 2017

To help you query logs more effectively, Log Service provides a set of query syntax to express query conditions. You can specify query conditions by using the GetLogs and GetHistograms interfaces in Log Service API or on the query page of the Log Service console. This document introduces the syntax of query conditions in details.

Index types

Log Service supports creating an index for the LogStore in the following methods:

  • Full text index: Query the entire line of logs as a whole without differentiating key and value.
  • Key/value index: Query logs after specifying a key. For example, FILE:app and Type:action. All the strings under the specified key will be queried.

Syntax keywords

LogSearch query conditions support the following keywords.

Name Meaning
and Binary operator. Format: query1 and query2. Indicates the intersection of the query results of query1 and query2. With no syntax keyword among multiple words, the relation is and by default.
or Binary operator. Format: query1 or query2. Indicates the union of the query results of query1 and query2.
not Binary operator. Format: query1 not query2. Indicates a result that meets query1 and does not meet query2, which is equivalent to query1–query2. If only not query1 exists, it indicates to select the results excluding query1 from all the logs.
(,) Parentheses () are used to merge one or more sub-queries into one query to increase the priority of the query in the parentheses ().
: Used to query the key-value pairs. term1:term2 makes up a key-value pair. If the key or value contains reserved characters such as spaces and colons (:), use quotation marks (“) to enclose the entire key or value.
Converts a keyword to a common query character. Each term enclosed in quotation marks (“) can be queried and will not be considered as a syntax keyword. Or all the terms enclosed in quotation marks (“) are regarded as a whole in the key-value query.
\ Escape character. Used to escape quotation marks (“). After being escaped, the quotation marks (“) indicate the symbols themselves and will not be considered as escape characters. For example, "\"".
| Pipeline operator. Indicates to perform more computing based on the previous computing. For example, query1 | timeslice 1h | count.
timeslice Time slice operator. Indicates the length of time during which the data is regarded as a whole for computing. The usage modes are timeslice 1h, timeslice 1m, and timeslice 1s, respectively indicating to regard 1 hour, 1 minute, and 1 second as a whole. For example, query1 | timeslice 1h | count indicates to query the query condition and return the total number of times with 1 hour as the time slice.
count Count operator. Indicates the number of log entries.
* Fuzzy query keyword. Used to replace zero or multiple characters. For example, the query results of que* start with que.
Note: At most 100 query results can be returned.
? Fuzzy query keyword. Used to replace one character. For example, the query results of qu?ry start with qu, end with ry, and have a character in the middle.
__topic__ Query the data under a topic. With the new syntax, you can query the data of zero or multiple topics in the query. For example, __topic__:mytopicname.
__tag__ Query a tag value under a tag key. For example, __tag__:tagkey:tagvalue.
source Query the data of an IP. For example, source:127.0.0.1.
> Query the logs with a field value greater than a specific number. For example, latency > 100.
>= Query the logs with a field value greater than or equal to a specific number. For example, latency >= 100.
< Query the logs with a field value less than a specific number. For example, latency < 100.
<= Query the logs with a field value less than or equal to a specific number. For example, latency <= 100.
= Query the logs with a field value equal to a specific number. For example, latency = 100.
in Query the logs with a field staying within a specific range. Brackets ([]) are used to indicate closed intervals and parentheses (()) are used to indicate open intervals. Enclose two numbers in brackets ([]) or parentheses (()) and separate the numbers with several spaces. For example, latency in [100 200] or latency in (100 200]. .

Note:

  • Syntax keywords are case-insensitive.
  • Priorities of syntax keywords are sorted in descending order as follows: : > " > ( ) > and not > or.
  • Log Service reserves the right to use the following keywords: sort asc desc group by avg sum min max limit. To use these keywords, enclose them in quotation marks (“).
  • If both the full text index and key/value index are configured and have different word segmentation characters, data cannot be queried using the full text query method.
  • Set the column type as double or long before performing a numeric query. If the column type is not set or the syntax used for the numeric range query is incorrect, Log Service translates the query condition as a full text index, which might lead to an unexpected result.
  • If you change the column type from text to numeric, only the = query is supported for the data before this change.

Query examples

  • Logs that contain a and b at the same time: a and b or a b.
  • Logs that contain a or b: a or b.
  • Logs that contain a but do not contain b: a not b.
  • All the logs that do not contain a: not a.
  • Query the logs that contain a and b, but do not contain c: a and b not c.
  • Logs that contain a or b and must contain c: (a or b ) and c.
  • Logs that contain a or b, but do not contain c: (a or b ) not c.
  • Logs that contain a and b and might contain c: a and b or c.
  • Logs whose FILE field contains apsara: FILE:apsara.
  • Logs whose FILE field contains apsara and shennong: FILE:"apsara shennong", FILE:apsara FILE: shennong, or FILE:apsara and FILE:shennong.
  • Logs containing and: and.
  • Logs with the FILE field containing apsara or shennong: FILE:apsara or FILE:shennong.
  • Logs with the file info field containing apsara: "file info":apsara.
  • Logs that contain quotation marks (“): \".
  • Query all the logs starting with shen: shen*.
  • Query all the logs starting with shen under the FILE field: FILE:shen*.
  • Query all the logs starting with shen, ending with ong, and having a character in the middle: shen?ong.
  • Query the logs starting with shen and aps: shen* and aps*.
  • Query the distribution of logs starting with shen, with 20 minutes as the time slice: shen*| timeslice 20m | count.
  • Query all the data under topic1 and topic2: __topic__:topic1 or __topic__ : topic2.
  • Query all the data of tagvalue2 under tagkey1: __tag__ : tagkey1 : tagvalue2.
  • Query all the data with a latency greater than or equal to 100 and less than 200, which can be written in the following two ways: latency >=100 and latency < 200 or latency in [100 200).
  • Query all the requests with a latency greater than 100, which must be written in the following way: latency > 100.
  • Query the logs that do not contain spider and do not contain opx in http_referer: not spider not bot not http_referer:opx.

Other syntaxes

Specified or cross-topic query

Each LogStore can be divided into one or more subspaces by the topic. During query, specifying topics can limit the query range so as to increase the speed. Therefore, we recommend that you use topic to divide the LogStore if you have a secondary classification requirement for the LogStore.

With one or more topics specified, the query is only performed in the topics that meet the conditions. However, data under all the topics is queried by default if no topic is specified.

For example, use topic to classify logs under different domain names:

topic

Topic query syntax:

  • Data under all the topics can be queried. Data of all the topics will be queried if no topic is specified in the query syntax and parameter.
  • Topic can be queried in the query. The query syntax is __topic__:topicName. The old mode (specify the topic in the URL parameter) is still supported at the same time.
  • Multiple topics can be queried. For example, __topic__:topic1 or __topic__:topic2 indicates to query the data union under Topic1 and Topic2.
Thank you! We've received your feedback.