This topic describes how to use the search syntax that is provided by Log Service to configure search conditions. You can query logs based on the search syntax with high efficiency.

Search types

A search statement specifies one or more search conditions and returns the logs that meet the specified conditions. Searches are classified by indexing method into full-text searches and field-specific searches, or classified by precision into exact searches and fuzzy searches.
Note
  • If you configure both full-text indexes and field indexes, the configurations of the field indexes take precedence.
  • Before you can specify a numeric range to query logs based on a field, you must set the data type of the field to double or long. If you do not set the data type of a field to double or long, or the syntax of the numeric range is invalid, Log Service performs a full-text search and the search result may be different from the expected result. For example, if you execute the owner_id>100 search statement and the data type of the owner_id field is not double or long, logs that contain owner_id, > (non-delimiter), and 100 are returned.
  • If you change the data type of a field from text to double or long, you can use only the equal sign (=) to query the logs that are collected before the change.
  • Full-text searches and field-specific searches
    Search type Description Example
    Full-text search After you configure full-text indexes, Log Service splits a log into multiple words by using the delimiters that you specify. You can specify keywords and rules in a search statement to query logs. The keywords can be field names or field values. PUT and cn-shanghai: returns the logs that contain the keywords PUT and cn-shanghai.
    Field-specific search After you configure field indexes, you can query logs. To query logs specify field names and field values in the key:value format. You can perform basic searches or combined searches based on the data types of the fields in the field indexes. For more information, see Data types. request_time>60 and request_method:Ge*: returns the logs in which the value of the request_time field is greater than 60 and the value of the request_method field starts with Ge.
  • Exact searches and fuzzy searches
    Search type Description Example
    Exact search Complete words are used for queries.
    • host:example.com: returns the logs in which the value of the host field is example.com.
    • PUT: returns the logs that contain the keyword PUT.
    Fuzzy search You can add an asterisk (*) or a question mark (?) as a wildcard in the middle or at the end of a keyword to perform a fuzzy search. Each keyword must be 1 to 64 characters in length. If a keyword contains a wildcard, Log Service searches all logs and obtains up to 100 words that match the keyword. Then, Log Service returns logs that contain one or more of these words. The more accurate a keyword is, the more accurate the search results are.
    Note
    • A keyword cannot start with an asterisk (*) or a question mark (?).
    • The long and double data types do not support asterisks (*) or question marks (?) in fuzzy searches. You can specify a numeric range when you perform a fuzzy search. Example: status in [200 299].
    A fuzzy search is performed based on samples by using the following mechanism:
    • If you enable the field indexing feature and specify a field to query logs, Log Service randomly obtains samples from the indexed data of the field and returns part of the search results.
    • If you enable the full-text indexing feature and do not specify a field to query logs, Log Service randomly obtains samples from the full-text indexed data and returns part of the search results.
    • addr*: searches for 100 words that start with addr from logs, and returns the logs that contain one or more of these words.
    • host:www.yl*: searches for 100 words that start with www.yl from the value of the host field. Then, Log Service returns the logs in which the value of the host field contains one or more of these words.
    For more information, see How do I query logs by using fuzzy match?

Operators

The following table describes the operators that are supported for search statements.
Note
  • The in operator is case-sensitive. Other operators are not case-sensitive.
  • Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use these operators as keywords, you must enclose the operators in double quotation marks ("").
  • The following list shows the priorities of operators in descending order:
    1. Colons (:)
    2. Double quotation marks ("")
    3. Parentheses ()
    4. and and not
    5. or
Operator Description
and The and operator. Example: request_method:GET and status:200.

If no syntax keywords exist among multiple keywords, the keywords are evaluated by using the and operator. For example, GET 200 cn-shanghai is equivalent to GET and 200 and cn-shanghai.

or The or operator. Example: request_method:GET or status:200.
not The not operator. Example: request_method:GET not status:200 or not status:200.
( ) This operator is used to increase the priority of the query conditions that are enclosed in parentheses (). Example: (request_method:GET or request_method:POST) and status:200.
: This operator is used for field-specific searches based on the key:value format. Example: request_method:GET.

If a field name or a field value contains reserved characters such as spaces and colons (:), you must enclose the field name or field value in double quotation marks (""). Example: "file info":apsara.

"" This operator is used to enclose a syntax keyword. If a syntax keyword is enclosed in double quotation marks (""), the keyword is converted to an ordinary character. For example, "and" returns the logs that contain and. In this case, and is not an operator.

In a field-specific search, the words that are enclosed in double quotation marks ("") are considered as a whole.

\ The escape character. This character is used to escape double quotation marks (""). Double quotation marks ("") can indicate themselves only after they are escaped. For example, if the content of a log is instance_id:nginx"01", you can execute the instance_id:nginx\"01\" statement to search for the log.
* The wildcard character. This character is used to match zero, one, or multiple characters. Example: host:aliyund*c.
Note Log Service searches all logs and obtains up to 100 words that match the specified conditions. Then, Log Service returns the logs that contain one or more of these words and match the query conditions.
? The wildcard character. This character is used to match a single character. Example: host:aliyund?c.
> This operator is used to query the logs in which the value of a specified field is greater than a specified numeric value. Example: request_time>100.
>= This operator is used to query the logs in which the value of a specified field is greater than or equal to a specified numeric value. Example: request_time>=100.
< This operator is used to query the logs in which the value of a specified field is smaller than a specified numeric value. Example: request_time<100.
<= This operator is used to query the logs in which the value of a specified field is smaller than or equal to a specified numeric value. Example: request_time<=100.
= This operator is used to query the logs in which the value of a specified field is equal to a specified numeric value. Equal signs (=) and colons (:) have the same effect on fields of the double or long data type. For example, request_time=100 is equivalent to request_time:100.
in This operator is used to query the logs in which the value of a specified field is within a specified numeric range. Brackets [] indicate a closed interval, and parentheses () indicate an open interval. A space is used to separate two numbers in a numeric range. Examples: request_time in [100 200] and request_time in (100 200].
Note The characters of in must be in lowercase.
__source__ This operator is used to query the logs of a specified log source. Wildcard characters are supported. Example: __source__:192.0.2.*.
Important The __source__ field is a reserved field in Log Service. This field can be abbreviated to source. If you configure a custom source field, the custom field conflicts with the reserved field source in Log Service. If you want to search for the custom field, you must use Source or SOURCE in the search statement.
__tag__ This operator is used to query logs by using metadata. Example: __tag__:__receive_time__:1609837139.
__topic__ This operator is used to query the logs of a specified log topic. Example: __topic__:nginx_access_log.

Examples of search statements

If you execute a search statement on different logs based on different index configurations, the statement returns different results. The examples in this section are provided based on the following sample log and index configurations.

Sample log

An NGINX access log is used as the sample log.

Sample log

Index configurations

Before you can execute a search statement, make sure that indexes are configured. For more information, see Configure indexes. Indexes

Common search examples

Expected search result Search statement
Logs that record successful GET requests (status codes: 200 to 299) 
request_method:GET and status in [200 299]
Logs that record GET requests and in which the source region of the requests is not the China (Hangzhou) region 
request_method:GET not region:cn-hangzhou
Logs that record GET requests or POST requests 
request_method:GET or request_method:POST
Logs that do not record GET requests 
not request_method:GET
Logs that record successful GET requests or successful POST requests 
(request_method:GET or request_method:POST) and status in [200 299]
Logs that record failed GET requests or failed POST requests 
(request_method:GET or request_method:POST) not status in [200 299]
Logs that record successful GET requests (status codes: 200 to 299) and in which the request duration is less than 60 seconds 
request_method:GET and status in [200 299] not request_time>=60
Logs in which the request duration is equal to 60 seconds 
  • request_time:60
  • request_time=60
Logs in which the request duration is greater than or equal to 60 seconds and is less than 200 seconds 
  • request_time>=60 and request_time<200
  • request_time in [60 200)
Logs in which the request_time field is empty or the value of the field is an invalid number 
request_time:* not request_time > -10000000000
Logs that contain the request_time field and in which the value of the field is a number 
request_time > -1000000000
Logs that contain and 
"and"
Note In this search statement, and is a common string but not an operator.
Logs in which the value of the request method field contains PUT 
"request method":PUT
Note The name of the request method field contains spaces. You must enclose the field name in double quotation marks ("") in a search statement.
Logs whose topic is HTTPS or HTTP 
__topic__:HTTPS or __topic__:HTTP
Logs that are collected from the 192.0.2.1 host 
__tag__:__client_ip__:192.0.2.1
Note The __tag__:__client_ip__ field is a reserved field in Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields.

If a log is processed by using the data transformation feature or a Logtail plug-in, the key in a tag field is converted to a common key. If you want to search for the log, you must enclose the name of the tag field in double quotation marks ("") in the search statement. Example: "__tag__:__client_ip__":192.0.2.1.

Logs in which the remote_user field is not empty 
not remote_user:""
Logs in which the remote_user field is empty 
remote_user:""
Logs in which the value of the remote_user field is not null 
not remote_user:"null"
Logs that do not contain the remote_user field 
not remote_user:*
Logs that contain the remote_user field 
remote_user:*
Logs in which the value of the city field is not Shanghai 
not city:Shanghai

Advanced search examples

  • Fuzzy searches
    Expected search result Search statement
    Logs that contain certain words. The words start with cn. 
    cn*
    Logs in which the value of the region field starts with cn. 
    region:cn*
    Logs in which the value of the region field contains cn*. 
    region:"cn*"
    Note In this search statement, cn* is a complete word. Examples:
    • If the content of a log is region:cn*,en and the delimiter is a comma (,), Log Service splits the log content into region, cn*, and en. You can use the search statement to search for the log.
    • If the content of a log is region:cn*hangzhou, Log Service considers cn*hangzhou as a whole. Therefore, you cannot use the search statement to search for the log.
    Logs that contain certain words. The words start with mozi, end with la, and include one character between mozi and la. 
    mozi?la
    Logs that contain certain words. The words start with mo, end with la, and include zero, one, or more characters between mo and la. 
    mo*la
    Logs that contain certain words. The words start with moz or sa. 
    moz* and sa*
    Logs in which the value of the region field ends with hai. You cannot use a search statement to search for the logs. You can use the LIKE clause in an SQL statement to search for the logs. For more information, see Use the LIKE clause to implement fuzzy match. 
    * | select * from log where region like '%hai'
  • Delimiter-based searches
    Log Service splits the content of a log into multiple words based on the delimiters that you specify. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If you leave Delimiter empty, Log Service considers the value of each field as a whole. In this case, you can search for a log only by using a complete string or a fuzzy search. For more information about how to specify delimiters, see Configure indexes.
    Note If a search keyword contains delimiters, you can perform a phrase search or use a LIKE clause. Examples:
    • Phrase search: #"redo_index/1". For more information, see Phrase search.
    • LIKE clause: * | select * from log where key like 'redo_index/1'.
    For example, the value of the http_user_agent field is Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/192.0.2.0 Safari/537.2.
    • If you leave Delimiter empty, Log Service considers the field value as a whole. In this case, you cannot search for logs by using the http_user_agent:Chrome search statement.
    • If you specify a value for Delimiter, Log Service splits the field value into Mozilla, 5.0, Windows, NT, 6.1, AppleWebKit, 537.2, KHTML, like, Gecko, Chrome, 192.0.2.0, Safari, and 537.2. The delimiters are , '";=()[]{}?@&<>/:\n\t\r. In this case, you can search for logs by using the http_user_agent:Chrome search statement.
    Expected search result Search statement
    Logs in which the value of the http_user_agent field contains Chrome 
    http_user_agent:Chrome
    Logs in which the value of the http_user_agent field contains Linux and Chrome 
    • http_user_agent:"Linux Chrome"
    • http_user_agent:Linux and http_user_agent:Chrome
    Logs in which the value of the http_user_agent field contains Firefox or Chrome 
    http_user_agent:Firefox or http_user_agent:Chrome
    Logs in which the value of the request_uri field contains /request/path-2 
    request_uri:/request/path-2
    Logs in which the value of the request_uri field starts with /request and does not contain /file-0 
    request_uri:/request* not request_uri:/file-0
  • Searches for the JSON logs whose field values are JSON objects or JSON arrays

    If the value of a field is in the JSON format, you can set the data type of the field to JSON or use JSON functions for query and analysis. For more information, see FAQ about the query and analysis of JSON logs.