This topic describes the search syntax that is used in Log Service.

Search types

After you enable the indexing feature and create indexes for the data in a Logstore, you can enter a search statement on the search and analysis page to filter logs.

A search statement specifies one or more search conditions and returns the log entries that match the search conditions. You can execute a search statement to perform a full-text search or field-specific search.
  • Full-text search

    During full-text search, a log entry is considered as a key-value pair. The value in the key-value pair indicates the content of the log entry. A full-text search statement returns the log entries that include or exclude the keywords as specified.

    Full-text search is divided into basic full-text search, phrase search, and fuzzy search.
    • Basic full-text search: You can specify keywords and rules in search conditions of a search statement. You can then execute the search statement to query the log entries that match the search conditions.

      For example, the a and b statement returns the log entries that include the a and b keywords.

    • Phrase search: A phrase is a string that is enclosed in double quotation marks (""). Substrings in a phrase are separated by space characters. Each substring is a keyword.

      For example, the msg:"http error" statement returns the log entries whose msg field contains the http and error keywords. This statement is equivalent to msg:http and msg:error.

    • Fuzzy search: You can use an asterisk (*) or a question mark (?) as a fuzzy search keyword. Each keyword can contain 1 to 64 characters in length and cannot start with the asterisk (*) or the question mark (?). If a search condition contains a fuzzy search keyword, Log Service returns a maximum of 100 log entries and each log entry contains a word that matches the keyword pattern. For example, if you execute the addr? statement, Log Service returns a maximum of 100 log entries and each log entry contains a word that starts with addr. For more information about how to use fuzzy search, see Fuzzy match.
      Fuzzy search is a sample-based search that uses the following mechanism:
      • If you enable the field indexing feature and specify a search field, Log Service randomly samples from the index data of the specified field and returns part of the search results.
      • If you enable the full-text indexing feature and do not specify a search field, Log Service randomly samples from the full-text index data and returns part of the search results.
      Note
      Fuzzy search has the following limits:
      • A keyword cannot start with an asterisk (*) or a question mark (?).
      • The more accurate a keyword is, the more accurate the query results will be.
      • A keyword cannot exceed 64 characters in length.
      • When you execute a fuzzy search statement, Log Service returns a maximum of 100 log entries that match the search conditions.
  • Field-specific search

    After you configure field indexes, you can search for log entries based on the keys and values of the fields in the field indexes. For a field of the double or long type, you can specify a value range in search statements. For example, after you execute the Latency>5000 and Method:Get* and not Status:200 statement, Log Service returns the log entries that meet the following conditions: The value of the Latency field is greater than 5000, the value of the Method field starts with Get, and the value of the Status field is not 200.

    You can perform a basic query or combined query based on the data types of the fields in the field indexes. For more information, see Data types of indexes.

Usage notes

  • If you execute a search statement to perform both full-text search and field-specific search, and you set different delimiters for the two search types, the delimiter for field-specific search prevails.
  • You must set the data type of a field to double or long before you specify a value range to search the field. If the data type of a field is not double or long, or the value range syntax is invalid, the field-specific search condition is considered a full-text search condition. In this case, the result of the query may be different from the result you expect.
  • If you change the data type of a field from text to double or long, only the equal-to operator (=) can be used to search for the log entries that are collected before the change.

Operators

The following table lists the operators that are supported by search statements.
Operator Description
and A binary operator. The syntax is query1 and query2. It indicates the intersection of the search results of query1 and query2. If no syntax keyword exists among the words in a search statement, the relationship of these words is and by default.
or A binary operator. The syntax is query1 or query2. It indicates the union of the query results of query1 and query2.
not A binary operator. The syntax is query1 not query2. It indicates that the log entries that match query1 but do not match query2 are returned. The syntax is equivalent to query1–query2. If a query statement is not query1, the log entries that do not matchquery1 are returned.
( , ) The operator that merges one or more sub-conditions into one search condition. If you enclose a sub-condition in parentheses (), the search based on the sub-condition is performed first.
: The operator that is used to specify a pattern of key-value pairs. The syntax is term1:term2. If the key or value contains reserved characters such as spaces and colons (:), use double quotation marks ("") to enclose the entire key or value.
" The operator that converts a keyword into a common character. All terms enclosed in double quotation marks ("") are considered keywords rather than operators. You can also enclose the entire key or value in double quotation marks.
\ The operator that escapes a double quotation mark. The escaped double quotation mark is considered a symbol rather than an operator. Example: "\"".
| The pipeline operator indicates that another calculation is performed based on the previous calculation. Example: query1 | select count (1).
* A fuzzy search character that is used to replace zero or more characters. For example, after you execute the que* statement, Log Service returns the log entries with a word that starts with que.
Note After you execute a fuzzy search statement, Log Service returns a maximum of 100 log entries that match the search condition.
? A fuzzy search character that is used to replace a single character. For example, if you use qu?ry in a search statement, the log entries with words that start with qu, end with ry, and contain a character in between are returned.
__topic__ The operator that can query logs for a specific topic. You can query zero or more topics. Example: __topic__:mytopicname.
__tag__ The operator that specifies a tag value of a tag key. Example: __tag__:tagkey:tagvalue.
source The operator that specifies the IP address of a log source whose log entries you want to query. Example: source:127.0.0.1.
> Queries the logs whose value of a field is greater than a specific number. Example: latency > 100.
>= Queries the logs whose value of a field is greater than or equal to a specific number. Example: latency >= 100.
< Queries the logs whose value of a field is less than a specific number. Example: latency < 100.
<= Queries the logs whose value of a field is less than or equal to a specific number. Example: latency <= 100.
= Queries the logs whose value of a field is equal to a specific number. Example: latency = 100.
in Queries the logs whose value of a field falls into a specific range. Brackets [] indicate closed intervals and parentheses () indicate open intervals. Two numbers are enclosed in brackets or parentheses and are separated by one or more spaces. The in operator must be in lowercase. Example: latency in [100 200] or latency in (100 200].
Note
  • All operators except the in operator are not case-sensitive.
  • You can use the following operators, which are sorted in descending order of precedence: :, ", (), and, not, and or.
  • Log Service uses the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you need to use these operators as keywords, enclose them in double quotation marks ("").

Search statement examples

Expected search result Search statement
Log entries that contain a and b a and b or a b
Log entries that contain a or b a or b
Log entries that contain a but do not contain b a not b
Log entries that do not contain a not a
Log entries that contain a and b but do not contain c a and b not c
Log entries that contain a or b and contain c (a or b ) and c
Log entries that contain a or b but do not contain c (a or b ) not c
Log entries that contain a and b and may contain c a and b or c
Log entries whose FILE field contains apsara FILE:apsara
Log entries whose FILE field contains apsara and shennong FILE:"apsara shennong" or FILE:apsara and FILE:shennong
Log entries that contain the keyword and and"
Log entries whose FILE field contains apsara or shennong FILE:apsara or FILE:shennong
Logs whose file info field contains apsara "file info":apsara
Log entries that contain double quotation marks (") \"
Log entries that start with shen shen*
Log entries whose FILE field starts with shen FILE:shen*
Log entries whose FILE field ends with shen Not supported
Log entries whose value of the FILE field is shen* FILE: "shen*"
Log entries with words that start with shen, end with ong, and contain a single character in between shen? ong
Log entries with words that start with shen, end with ong, and contain multiple characters in between shen*ong
Log entries with words that start with shen and words that start with aps shen* and aps*
Log entries of topic1 and topic2 __topic__:topic1 or __topic__ : topic2
Log entries with a tag whose key is tagkey1 and value is tagvalue2 __tag__ : tagkey1 : tagvalue2
Log entries whose value of the latency field is greater than or equal to 100 and less than 200 latency >=100 and latency < 200 or latency in [100 200)
Log entries whose value of the latency field is greater than 100 latency > 100
Log entries that do not contain spiders or bots, and whose http_referer field does not contain opx not spider not bot not http_referer:opx
Log entries whose cdnIP field is not empty not cdnIP:""
Log entries that do not contain the cdnIP field not cdnIP:*
Log entries that contain the cdnIP field cdnIP:*
Log entries that contain a specified URL * | select * where url = 'www.xxxxx.com'

Topic-specific search

Each Logstore can be divided into one or more topics. You can divide a Logstore into multiple topics if you have a secondary classification requirement for log entries. When you query logs, you can specify topics to increase efficiency.

In a search statement, you can specify one or more topics. If no topic is specified, log entries are queried from all topics.

For example, you can classify log entries into multiple topics based on domain names.Log topics
Syntax of topic-specific search:
  • In a search statement, you can specify one or more topics. If no topic is specified, log entries are queried from all topics.
  • The topic-specific search syntax is __topic__:topicName.
  • You can query log entries from multiple topics. For example, the __topic__:topic1 or __topic__:topic2 statement returns the log entries in topic1 and topic2.