All Products
Search
Document Center

Simple Log Service:Search syntax

Last Updated:Jan 19, 2024

This topic describes how to use the search syntax that is provided by Simple Log Service to specify search conditions. You can efficiently query logs based on the search syntax.

Search types

A search statement specifies one or more search conditions and returns the logs that meet the specified conditions. Searches are classified by indexing method into full-text searches and field-specific searches, or classified by precision into exact searches and fuzzy searches.

Note
  • Before you can specify a numeric range to query logs based on a field, you must set the data type of the field to double or long. If you do not set the data type of a field to double or long, or the syntax of the numeric range is invalid, Simple Log Service performs a full-text search and the search result that is returned may be different from the expected result. For example, if you execute the owner_id>100 search statement and the data type of the owner_id field is not double or long, logs that contain owner_id, >, and 100 are returned. In this example, the greater sign (>) is not a delimiter.

  • If you change the data type of a field from text to double or long, you can use only the equal sign (=) to query the logs that are collected before the change.

  • Query and analysis results vary based on index configurations. You must configure indexes based on your business requirements. If you create both full-text indexes and field indexes, the configurations of the field indexes take precedence.

  • Full-text searches and field-specific searches

    Search type

    Description

    Example

    Full-text search

    After you configure full-text indexes, Simple Log Service splits a log into multiple strings by using the delimiters that you specify. You can specify keywords and rules in a search statement to query logs. The keywords can be field names or field values.

    PUT and cn-shanghai: returns the logs that contain the keywords PUT and cn-shanghai.

    Field-specific search

    After you configure field indexes, you can query logs. To query logs, specify field names and field values in the key:value format. You can perform basic searches or combined searches based on the data types of the fields in the field indexes. For more information, see Data types.

    request_time>60 and request_method:Ge*: returns the logs in which the value of the request_time field is greater than 60 and the value of the request_method field starts with Ge.

  • Exact searches and fuzzy searches

    Search type

    Description

    Example

    Exact search

    Complete strings are used for queries.

    Simple Log Service uses word segmentation to query logs. Phrases cannot be completely matched based on an exact search. For example, the search statement abc def returns the logs that contain abc or def. The phrase abc def cannot be completely matched. If you want the phrase abc def to be completely matched, you can perform a phrase search or use a LIKE clause. For more information, see Phrase search and How do I query logs by using exact match?

    • host:example.com: returns the logs in which the value of the host field contains example.com.

    • PUT: returns the logs that contain the keyword PUT.

    • * | Select * where http_user_agent like '%like Gecko%': returns the logs in which the value of the http_user_agent field contains the phrase like Gecko.

    • #"redo_index/1": returns the logs that contain the phrase redo_index/1.

    Fuzzy search

    You can add an asterisk (*) or a question mark (?) as a wildcard character in the middle or at the end of a keyword to perform a fuzzy search. Each keyword must be 1 to 64 characters in length. If a keyword contains a wildcard character, Simple Log Service searches all logs and obtains up to 100 strings that match the keyword. Then, Simple Log Service returns the logs that contain one or more of these strings. The more accurate a keyword is, the more accurate the search results are.

    Important
    • A keyword cannot start with an asterisk (*) or a question mark (?).

    • The long and double data types do not support asterisks (*) or question marks (?) in fuzzy searches. You can specify a numeric range when you perform a fuzzy search. Example: status in [200 299].

    A fuzzy search is performed based on samples by using the following mechanism:

    • If you enable the field indexing feature and specify a field to query logs, Simple Log Service randomly obtains samples from the indexed data of the field and returns part of the search results.

    • If you enable the full-text indexing feature and do not specify a field to query logs, Simple Log Service randomly obtains samples from the full-text indexed data and returns part of the search results.

    • addr*: searches for 100 strings that start with addr from logs, and returns the logs that contain one or more of these strings.

    • host:www.yl*: searches for 100 strings that start with www.yl from the value of the host field. Then, Simple Log Service returns the logs in which the value of the host field contains one or more of these strings.

    For more information, see How do I query logs by using fuzzy match?.

Operators

The following table describes the operators that are supported by search statements.

Important
  • The in operator is case-sensitive. Other operators are not case-sensitive.

  • Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use the preceding operators as keywords, you must enclose the operators in double quotation marks ("").

  • The following list shows the priorities of the operators in descending order:

    1. Colons (:)

    2. Double quotation marks ("")

    3. Parentheses ()

    4. and and not

    5. or

Operator

Description

and

The and operator. Example: request_method:GET and status:200.

If no syntax keyword exists among multiple keywords, the keywords are joined by using the and operator by default. For example, GET 200 cn-shanghai is equivalent to GET and 200 and cn-shanghai.

or

The or operator. Example: request_method:GET or status:200.

not

The not operator. Example: request_method:GET not status:200 or not status:200.

( )

This operator is used to increase the priority of the search conditions that are enclosed in parentheses (). Example: (request_method:GET or request_method:POST) and status:200.

:

This operator is used for field-specific searches based on the key:value format. Example: request_method:GET.

If a field name or a field value contains reserved characters such as space characters and colons (:), you must enclose the field name or field value in double quotation marks (""). Example: "file info":apsara.

""

This operator is used to enclose a syntax keyword. If a syntax keyword is enclosed in double quotation marks (""), the keyword is converted to an ordinary character. For example, "and" returns the logs that contain and. In this case, and is not an operator.

In a field-specific search, the strings that are enclosed in double quotation marks ("") are considered as a whole string.

\

The escape character. This character is used to escape double quotation marks (""). Double quotation marks ("") can indicate themselves only after they are escaped. For example, if the content of a log is instance_id:nginx"01", you can execute the instance_id:nginx\"01\" statement to search for the log.

*

The wildcard character. This character is used to match zero, one, or multiple characters. Example: host:aliyund*c.

Note

Simple Log Service searches all logs and obtains up to 100 strings that meet the specified conditions. Then, Simple Log Service returns the logs that contain one or more of the 100 strings and meet the search conditions.

?

The wildcard character. This character is used to match a single character. Example: host:aliyund?c.

>

This operator is used to query the logs in which the value of a specified field is greater than a specified numeric value. Example: request_time>100.

>=

This operator is used to query the logs in which the value of a specified field is greater than or equal to a specified numeric value. Example: request_time>=100.

<

This operator is used to query the logs in which the value of a specified field is smaller than a specified numeric value. Example: request_time<100.

<=

This operator is used to query the logs in which the value of a specified field is smaller than or equal to a specified numeric value. Example: request_time<=100.

=

This operator is used to query the logs in which the value of a specified field is equal to a specified numeric value. Equal signs (=) and colons (:) have the same effect on fields of the double or long data type. For example, request_time=100 is equivalent to request_time:100.

in

This operator is used to query the logs in which the value of a specified field is within a specified numeric range. Brackets [] indicate a closed interval, and parentheses () indicate an open interval. A space character is used to separate two numbers in a numeric range. Examples: request_time in [100 200] and request_time in (100 200].

Important

The characters of in must be in lowercase.

__source__

This operator is used to query the logs of a specified log source. Wildcard characters are supported. Example: __source__:192.0.2.*.

Important

The __source__ field is a reserved field in Simple Log Service. This field can be abbreviated to source. If you configure a custom field in the source format, the custom field conflicts with the reserved source field in Simple Log Service. If you want to search for the custom field, you must use Source or SOURCE in a search statement.

__tag__

This operator is used to query logs based on metadata. Example: __tag__:__receive_time__:1609837139.

__topic__

This operator is used to query the logs of a specified log topic. Example: __topic__:nginx_access_log.

Examples of search statements

If you execute a search statement on different logs based on different index configurations, the statement returns different results. The examples in this section are provided based on the following sample log and index configurations.

Sample log

An NGINX access log is used as the sample log.

Sample log

Index configurations

Before you can execute a search statement, make sure that indexes are configured. For more information, see Create indexes. Indexes

Common search examples

Expected search result

Search statement

Logs that contain successful GET requests (status codes: 200 to 299)

request_method:GET and status in [200 299]

Logs that contain GET requests but do not contain the China (Shanghai) region

request_method:GET not region:cn-hangzhou

Logs that contain GET requests or POST requests

request_method:GET or request_method:POST

Logs that do not contain GET requests

not request_method:GET

Logs that contain successful GET requests or successful POST requests

(request_method:GET or request_method:POST) and status in [200 299]

Logs that contain failed GET requests or failed POST requests

(request_method:GET or request_method:POST) not status in [200 299]

Logs that contain successful GET requests (status codes: 200 to 299) and in which the request duration is less than 60 seconds

request_method:GET and status in [200 299] not request_time>=60

Logs in which the request duration is equal to 60 seconds

  • request_time:60
  • request_time=60

Logs in which the request duration is greater than or equal to 60 seconds and is less than 200 seconds

  • request_time>=60 and request_time<200
  • request_time in [60 200)

Logs in which the request_time field is empty or the value of the field is an invalid number

request_time:* not request_time > -10000000000
Note

request_time:* is used to check whether the request_time field exists. In this search statement, the asterisk (*) does not indicate a fuzzy search.

Logs that contain the request_time field and in which the value of the field is a number

request_time > -1000000000

Logs that contain and

"and"
Note

In this search statement, and is a common string but not an operator.

Logs in which the value of the request method field contains PUT

"request method":PUT
Important

The name of the request method field contains a space character. You must enclose the field name in double quotation marks ("") in a search statement.

Logs whose topic is HTTPS or HTTP

__topic__:HTTPS or __topic__:HTTP

Logs that are collected from the 192.0.2.1 host

__tag__:__client_ip__:192.0.2.1

The __tag__:__client_ip__ field is a reserved field in Simple Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields.

Important

If a log is processed by using the data transformation feature or a Logtail plug-in, the key in a tag field is converted to a common key. If you want to search for the log, you must enclose the name of the tag field in double quotation marks ("") in the search statement. Example: "__tag__:__client_ip__":192.0.2.1.

Logs that contain 192.168.XX.XX

* | select * from log where key like '192.168.%.%'

For more information, see Use the LIKE clause to implement fuzzy match.

Logs in which the remote_user field is not empty

not remote_user:""

Logs in which the remote_user field is empty

remote_user:""

Logs in which the value of the remote_user field is not null

not remote_user:"null"

Logs that do not contain the remote_user field

not remote_user:*

Logs that contain the remote_user field

remote_user:*

Logs in which the value of the city field is not Shanghai

not city:Shanghai
Note

If you want to query Chinese strings, you must turn on Include Chinese when you configure indexes. For more information, see Create indexes.

Advanced search examples

  • Fuzzy search

    Expected search result

    Search statement

    Logs that contain specific strings. The strings start with cn.

    cn*

    Logs in which the value of the region field starts with cn.

    region:cn*

    Logs in which the value of the region field contains cn*.

    region:"cn*"
    Note

    In this search statement, cn* is a complete string. Examples:

    • If the content of a log is region:cn*,en and the delimiter is a comma (,), Simple Log Service splits the log content into region, cn*, and en. You can use the search statement to search for the log.

    • If the content of a log is region:cn*hangzhou, Simple Log Service considers cn*hangzhou as a whole. Therefore, you cannot use the search statement to search for the log.

    Logs that contain specific strings. The strings start with mozi, end with la, and include one character between mozi and la.

    mozi?la

    Logs that contain specific strings. The strings start with mo, end with la, and include zero, one, or more characters between mo and la.

    mo*la

    Logs that contain specific strings. The strings start with moz or sa.

    moz* and sa*

    Logs in which the value of the region field ends with hai.

    You cannot use a search statement to search for the logs. You can use the LIKE clause in an SQL statement to search for the logs. For more information, see Use the LIKE clause to implement fuzzy match.

    * | select * from log where region like '%hai'
  • Delimiter-based searches

    Simple Log Service splits the content of a log into multiple strings based on the delimiters that you specify. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If you leave the Delimiter parameter empty, Simple Log Service considers the value of each field as a whole. In this case, you can search for a log only by using a complete string or a fuzzy search. For more information about how to specify delimiters, see Create indexes.

    For example, the value of the http_user_agent field is Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/192.0.2.0 Safari/537.2.

    • If you leave the Delimiter parameter empty, Simple Log Service considers the field value as a whole. In this case, you cannot search for logs by using the http_user_agent:Chrome search statement.

    • If you set the Delimiter parameter to , '";=()[]{}?@&<>/:\n\t\r, Simple Log Service splits the field value into Mozilla, 5.0, Windows, NT, 6.1, AppleWebKit, 537.2, KHTML, like, Gecko, Chrome, 192.0.2.0, Safari, and 537.2. In this case, you can search for logs by using the http_user_agent:Chrome search statement.

    Note

    If a search keyword contains delimiters, you can perform a phrase search or use a LIKE clause. Examples:

    • Phrase search: #"redo_index/1". For more information, see Phrase search.

    • LIKE clause: * | select * from log where key like 'redo_index/1'.

    Expected search result

    Search statement

    Logs in which the value of the http_user_agent field contains Chrome

    http_user_agent:Chrome

    Logs in which the value of the http_user_agent field contains Linux and Chrome

    • http_user_agent:"Linux Chrome"
    • http_user_agent:Linux and http_user_agent:Chrome

    Logs in which the value of the http_user_agent field contains Firefox or Chrome

    http_user_agent:Firefox or http_user_agent:Chrome

    Logs in which the value of the request_uri field contains /request/path-2

    request_uri:/request/path-2

    Logs in which the value of the request_uri field starts with /request and does not contain /file-0

    request_uri:/request* not request_uri:/file-0

    Logs in which the phrase redo_index/1 is completely matched

    • #"redo_index/1"

    • * | select * from log where key like 'redo_index/1'

    Note

    You can perform a phrase search or use a LIKE clause to completely match the phrase. If you perform an exact search, strings such as redo_index and 1 are matched.

  • Searches for the JSON logs whose field values are JSON objects or JSON arrays

    If the value of a field is in the JSON format, you can set the data type of the field to JSON or use JSON functions to query and analyze the field. For more information, see FAQ about the query and analysis of JSON logs.