This topic describes the search syntax that is used in Log Service.

Search types

After you enable and configure the index feature of a Logstore, you can enter a search statement on the search and analysis page to filter logs.

A search statement specifies one or more search conditions and returns the log entries that match the search conditions. You can execute a search statement to perform a full-text search or field-specific search.
  • Full-text search

    During full-text search, a log entry is considered a key-value pair. The value in the key-value pair indicates the content of the log entry. A full-text search statement returns the log entries that include or exclude the specified keywords.

    Full-text search is divided into basic full-text search, phrase search, and wildcard-based search.
    • Basic full-text search: You can specify keywords and operators in search conditions of a search statement. You can then execute the search statement to query the log entries that match the search conditions.

      For example, the a and b statement returns the log entries that include the a and b keywords.

    • Phrase search: A phrase is a string that is enclosed in double quotation marks (""). Substrings in a phrase are separated by space characters. Each substring is a keyword.

      For example, the msg:"http error" statement returns the log entries whose msg field contains the http and error keywords. This statement is equivalent to msg:http and msg:error.

    • Wildcard-based search: You can use an asterisk (*) or a question mark (?) as a wildcard character in a keyword. Each keyword can contain 1 to 64 characters in length and cannot start with a wildcard character. If a search condition contains a keyword that includes a wildcard character, Log Service returns a maximum of 100 log entries and each log entry contains a word that matches the keyword pattern. For more information about how to use wildcard-based search, see Fuzzy match.
      For example, if you execute the addr? statement, Log Service returns a maximum of 100 log entries and each log entry contains a word that starts with addr.
      Note
      When you use wildcard-based search, note the following information:
      • A keyword cannot start with an asterisk (*) or a question mark (?).
      • The more accurate the keyword is, the more accurate the search results will be.
      • Wildcard-based search is not supported for a keyword that contains more than 64 characters in length.
      • A search statement returns a maximum of 100 log entries that match the search conditions.
  • Field-specific search

    After you configure the field index, you can search log entries based on the keys and values of the fields in the field index. For a field of the double or long type, you can specify a value range for search. For example, the Latency>5000 and Method:Get* and not Status:200 statement returns the log entries that meet the following conditions: The value of the Latency field is greater than 5000, the value of the Method field starts with Get, and the value of the Status field is not 200.

    You can perform a basic query or combined query, depending on the data types of the fields in the field index. For more information, see Indexes and data types.

Additional considerations

  • If you execute a search statement to perform both full-text search and field-specific search and you set different delimiters for the two search types, the delimiter that is set for field-specific search is used.
  • You must set the data type of a field to DOUBLE or LONG before you specify a value range to search the field. If the data type of a field is not DOUBLE or LONG or the value range syntax is incorrect, the field-specific search condition is considered a full-text search condition. In this case, unexpected search results may be returned.
  • If you change the data type of a field from TEXT to DOUBLE or LONG, only the equal-to operator (=) can be used to search for the log entries that are collected before the change.

Operators

The following table lists the operators that are supported by search statements.
Operator Description
and A binary operator. The syntax is query1 and query2. It indicates the intersection of the search results of query1 and query2. The default operator between keywords is and.
or A binary operator. The syntax is query1 or query2. It indicates the union of the search results of query1 or query2.
not A binary operator. The syntax is query1 not query2. It indicates that the log entries that match query1 but do not match query2 are returned. The syntax is equivalent to query1–query2. You can also use the not query1 syntax. It indicates that the log entries that do not match query1 are returned.
( , ) The operator that merges one or more sub-conditions into one search condition. The search based on a sub-condition that is enclosed in parentheses () is performed first.
: The operator that is used to specify a pattern of key-value pairs. The syntax is term1:term2. If the key or value contains reserved characters such as spaces and colons (:), use double quotation marks ("") to enclose the entire key or value.
" The operator that converts another operator into a common character. All terms enclosed in double quotation marks ("") are considered keywords rather than operators. In a field-specific search statement, you can enclose the entire key or value in double quotation marks.
\ The operator that escapes a double quotation mark. The escaped double quotation mark is considered a symbol instead of an operator. Example: "\"".
| The pipeline operator that is used to connect a search statement and an analytic statement. The analytic statement that follows the pipeline operator is performed based on the result of the search statement that the pipeline operator follows. Example: query1 | select count(1).
count The count operator that is used to summarize the number of log entries.
* The wildcard character that is used to replace zero or more characters. For example, the que* statement returns the log entries with a word that starts with que.
Note A wildcard-based search statement returns a maximum of 100 log entries that match the search condition.
? The wildcard character that replaces a single character. The qu?ry statement returns the log entries with a word that starts with qu, ends with ry, and contains a character in between.
__topic__ The operator that specifies zero or more topics from which to query log entries. Example: __topic__:mytopicname.
__tag__ The operator that specifies a tag value of a tag key to query. Example: __tag__:tagkey:tagvalue.
source The operator that specifies the IP address of a log source whose log entries you want to query. Example: source:127.0.0.1.
> The greater-than operator. You can use this operator to query the log entries whose value of a field is greater than a specified number. Example: latency > 100.
>= The greater-than-or-equal-to operator. You can use this operator to query the log entries whose value of a field is greater than or equal to a specified number. Example: latency >= 100.
< The less-than operator. You can use this operator to query the log entries whose value of a field is less than a specified number. Example: latency < 100.
<= The less-than-or-equal-to operator. You can use this operator to query the log entries whose value of a field is less than or equal to a specified number. Example: latency <= 100.
= The equal-to operator. You can use this operator to query the log entries whose value of a field is equal to a specified number. Example: latency = 100.
in The operator that is used to query the log entries whose value of a field falls in a specified range. Brackets [] indicate closed intervals and parentheses () indicate open intervals. The beginning number and ending number of the range are enclosed in brackets or parentheses and separated by one or more space characters. The in operator must be in lowercase. Example: latency in [100 200] or latency in (100 200].
Note
  • All operators except the in operator are case-insensitive.
  • You can use the following operators, which are sorted in descending order of precedence: :, ", (), and, not, and or.
  • Log Service uses the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you need to use these operators as keywords, enclose them in double quotation marks ("").

Search statement examples

Expected search result Search statement
Log entries that contain a and b a and b or a b
Log entries that contain a or b a or b
Log entries that contain a but do not contain b a not b
Log entries that do not contain a not a
Log entries that contain a and b but do not contain c a and b not c
Log entries that contain a or b and contain c (a or b ) and c
Log entries that contain a or b but do not contain c (a or b ) not c
Log entries that contain a and b and may contain c a and b or c
Log entries whose FILE field contains apsara FILE:apsara
Log entries whose FILE field contains apsara and shennong FILE:"apsara shennong" or FILE:apsara and FILE:shennong
Log entries that contain the following keyword: and and
Log entries whose FILE field contains apsara or shennong FILE:apsara or FILE:shennong
Logs whose file info field contains apsara "file info":apsara
Log entries that contain double quotation mark (") \"
Log entries that start with shen shen*
Log entries whose FILE field starts with shen FILE:shen*
Log entries whose value of the FILE field is shen* FILE: "shen*"
Log entries with words that start with shen, end with ong, and contain a single character in between shen?ong
Log entries with words that start with shen and words that start with aps shen* and aps*
Log entries of topic1 and topic2 __topic__:topic1 or __topic__ : topic2
Log entries with a tag whose key is tagkey1 and value is tagvalue2 __tag__ : tagkey1 : tagvalue2
Log entries whose value of the latency field is greater than or equal to 100 and less than 200 latency >=100 and latency < 200 or latency in [100 200)
Log entries whose value of the latency field is greater than 100 latency > 100
Log entries that do not contain spider and whose http_referer field does not contain opx not spider not bot not http_referer:opx
Log entries whose cdnIP field is not empty not cdnIP:""
Log entries that do not contain the cdnIP field not cdnIP:*
Log entries that contain the cdnIP field cdnIP:*
Log entries that contain a specified URL * | select * where url = 'www.xxxxx.com'

Topic-specific search

Each Logstore is divided into one or more topics. You can divide a Logstore into multiple topics if you need level-2 categories of log entries. When you query logs, you can specify topics to increase efficiency.

In a search statement, you can specify one or more topics to query. If no topic is specified, log entries are queried from all topics.

For example, you can classify log entries into multiple topics based on domain names.Log topics
Syntax of topic-specific search:
  • In a search statement, you can specify one or more topics to query. If no topic is specified, log entries are queried from all topics.
  • The topic-specific search syntax is __topic__:topicName.
  • You can query log entries from multiple topics. For example, the __topic__:topic1 or __topic__:topic2 statement returns the log entries in topic1 and topic2.