Search syntax - Log Service - Alibaba Cloud Documentation Center

This topic describes how to use the search syntax that is provided by Simple Log Service to specify search conditions. You can efficiently query logs based on the search syntax.

Search types

A search statement specifies one or more search conditions and returns the logs that meet the specified conditions. Searches are classified by indexing method into full-text searches and field-specific searches, or classified by precision into exact searches and fuzzy searches.

Note

Before you can specify a numeric range to query logs based on a field, you must set the data type of the field to double or long. If you do not set the data type of a field to double or long, or the syntax of the numeric range is invalid, Simple Log Service performs a full-text search and the search result that is returned may be different from the expected result. For example, if you execute the owner_id>100 search statement and the data type of the owner_id field is not double or long, logs that contain owner_id, >, and 100 are returned. In this example, the greater sign (>) is not a delimiter.
If you change the data type of a field from text to double or long, you can use only the equal sign (=) to query the logs that are collected before the change.
Query and analysis results vary based on index configurations. You must configure indexes based on your business requirements. If you create both full-text indexes and field indexes, the configurations of the field indexes take precedence.

Full-text searches and field-specific searches

Search type	Description	Example
Full-text search	After you configure full-text indexes, Simple Log Service splits a log into multiple strings by using the delimiters that you specify. You can specify keywords and rules in a search statement to query logs. The keywords can be field names or field values.	`PUT and cn-shanghai`: returns the logs that contain the keywords PUT and cn-shanghai.
Field-specific search	After you configure field indexes, you can query logs. To query logs, specify field names and field values in the key:value format. You can perform basic searches or combined searches based on the data types of the fields in the field indexes. For more information, see Data types.	`request_time>60 and request_method:Ge*`: returns the logs in which the value of the request_time field is greater than 60 and the value of the request_method field starts with Ge.

Exact searches and fuzzy searches

Search type

Description

Example

Exact search

Complete strings are used for queries.

Simple Log Service uses word segmentation to query logs. Phrases cannot be completely matched by based on an exact search. For example, the search statement abc def returns the logs that contain abc or def. The phrase abc def cannot be completely matched. If you want the phrase abc def to be completely matched, you can perform a phrase search or use a LIKE clause. For more information, see Phrase search and How do I query logs by using exact match?

host:example.com: returns the logs in which the value of the host field contains example.com.
PUT: returns the logs that contain the keyword PUT.
* | Select * where http_user_agent like '%like Gecko%': returns the logs in which the value of the http_user_agent field contains the phrase like Gecko.
#"redo_index/1": returns the logs that contain the phrase redo_index/1.

Fuzzy search

You can add an asterisk (*) or a question mark (?) as a wildcard character in the middle or at the end of a keyword to perform a fuzzy search. Each keyword must be 1 to 64 characters in length. If a keyword contains a wildcard character, Simple Log Service searches all logs and obtains up to 100 strings that match the keyword. Then, Simple Log Service returns the logs that contain one or more of these strings. The more accurate a keyword is, the more accurate the search results are.

Important

A keyword cannot start with an asterisk (*) or a question mark (?).
The long and double data types do not support asterisks (*) or question marks (?) in fuzzy searches. You can specify a numeric range when you perform a fuzzy search. Example: status in [200 299].

A fuzzy search is performed based on samples by using the following mechanism:

If you enable the field indexing feature and specify a field to query logs, Simple Log Service randomly obtains samples from the indexed data of the field and returns part of the search results.
If you enable the full-text indexing feature and do not specify a field to query logs, Simple Log Service randomly obtains samples from the full-text indexed data and returns part of the search results.

addr*: searches for 100 strings that start with addr from logs, and returns the logs that contain one or more of these strings.
host:www.yl*: searches for 100 strings that start with www.yl from the value of the host field. Then, Simple Log Service returns the logs in which the value of the host field contains one or more of these strings.

For more information, see How do I query logs by using fuzzy match?.

Operators

The following table describes the operators that are supported by search statements.

Important

The in operator is case-sensitive. Other operators are not case-sensitive.
Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use the preceding operators as keywords, you must enclose the operators in double quotation marks ("").
The following list shows the priorities of the operators in descending order:
1. Colons (:)
2. Double quotation marks ("")
3. Parentheses ()
4. and and not
5. or

Operator	Description
and	The and operator. Example: `request_method:GET and status:200`. If no syntax keyword exists among multiple keywords, the keywords are joined by using the and operator by default. For example, `GET 200 cn-shanghai` is equivalent to `GET and 200 and cn-shanghai`.
or	The or operator. Example: `request_method:GET or status:200`.
not	The not operator. Example: `request_method:GET not status:200` or `not status:200`.
( )	This operator is used to increase the priority of the search conditions that are enclosed in parentheses (). Example: `(request_method:GET or request_method:POST) and status:200`.
:	This operator is used for field-specific searches based on the key:value format. Example: `request_method:GET`. If a field name or a field value contains reserved characters such as space characters and colons (:), you must enclose the field name or field value in double quotation marks (""). Example: `"file info":apsara`.
""	This operator is used to enclose a syntax keyword. If a syntax keyword is enclosed in double quotation marks (""), the keyword is converted to an ordinary character. For example, `"and"` returns the logs that contain and. In this case, and is not an operator. In a field-specific search, the strings that are enclosed in double quotation marks ("") are considered as a whole string.
\	The escape character. This character is used to escape double quotation marks (""). Double quotation marks ("") can indicate themselves only after they are escaped. For example, if the content of a log is `instance_id:nginx"01"`, you can execute the `instance_id:nginx\"01\"` statement to search for the log.
*	The wildcard character. This character is used to match zero, one, or multiple characters. Example: `host:aliyundc`. Note* Simple Log Service searches all logs and obtains up to 100 strings that meet the specified conditions. Then, Simple Log Service returns the logs that contain one or more of the 100 strings and meet the search conditions.
?	The wildcard character. This character is used to match a single character. Example: `host:aliyund?c`.
>	This operator is used to query the logs in which the value of a specified field is greater than a specified numeric value. Example: `request_time>100`.
>=	This operator is used to query the logs in which the value of a specified field is greater than or equal to a specified numeric value. Example: `request_time>=100`.
<	This operator is used to query the logs in which the value of a specified field is smaller than a specified numeric value. Example: `request_time<100`.
<=	This operator is used to query the logs in which the value of a specified field is smaller than or equal to a specified numeric value. Example: `request_time<=100`.
=	This operator is used to query the logs in which the value of a specified field is equal to a specified numeric value. Equal signs (=) and colons (:) have the same effect on fields of the double or long data type. For example, `request_time=100` is equivalent to `request_time:100`.
in	This operator is used to query the logs in which the value of a specified field is within a specified numeric range. Brackets [] indicate a closed interval, and parentheses () indicate an open interval. A space character is used to separate two numbers in a numeric range. Examples: `request_time in [100 200]` and `request_time in (100 200]`. Important The characters of in must be in lowercase.
__source__	This operator is used to query the logs of a specified log source. Wildcard characters are supported. Example: `__source__:192.0.2.`. Important* The __source__ field is a reserved field in Simple Log Service. This field can be abbreviated to source. If you configure a custom field in the source format, the custom field conflicts with the reserved source field in Simple Log Service. If you want to search for the custom field, you must use Source or SOURCE in a search statement.
__tag__	This operator is used to query logs based on metadata. Example: `__tag__:__receive_time__:1609837139`.
__topic__	This operator is used to query the logs of a specified log topic. Example: `__topic__:nginx_access_log`.

Examples of search statements

If you execute a search statement on different logs based on different index configurations, the statement returns different results. The examples in this section are provided based on the following sample log and index configurations.

Sample log

An NGINX access log is used as the sample log.

Index configurations

Before you can execute a search statement, make sure that indexes are configured. For more information, see Create indexes. Indexes

Common search examples

Expected search result	Search statement
Logs that contain successful GET requests (status codes: 200 to 299)	`request_method:GET and status in [200 299]`
Logs that contain GET requests but do not contain the China (Shanghai) region	`request_method:GET not region:cn-hangzhou`
Logs that contain GET requests or POST requests	`request_method:GET or request_method:POST`
Logs that do not contain GET requests	`not request_method:GET`
Logs that contain successful GET requests or successful POST requests	`(request_method:GET or request_method:POST) and status in [200 299]`
Logs that contain failed GET requests or failed POST requests	`(request_method:GET or request_method:POST) not status in [200 299]`
Logs that contain successful GET requests (status codes: 200 to 299) and in which the request duration is less than 60 seconds	`request_method:GET and status in [200 299] not request_time>=60`
Logs in which the request duration is equal to 60 seconds	`request_time:60` `request_time=60`
Logs in which the request duration is greater than or equal to 60 seconds and is less than 200 seconds	`request_time>=60 and request_time<200` `request_time in [60 200)`
Logs in which the request_time field is empty or the value of the field is an invalid number	`request_time:* not request_time > -10000000000` Note `request_time:` is used to check whether the request_time field exists. In this search statement, the asterisk () does not indicate a fuzzy search.
Logs that contain the request_time field and in which the value of the field is a number	`request_time > -1000000000`
Logs that contain and	`"and"` Note In this search statement, and is a common string but not an operator.
Logs in which the value of the request method field contains PUT	`"request method":PUT` Important The name of the request method field contains a space character. You must enclose the field name in double quotation marks ("") in a search statement.
Logs whose topic is HTTPS or HTTP	`__topic__:HTTPS or __topic__:HTTP`
Logs that are collected from the 192.0.2.1 host	`__tag__:__client_ip__:192.0.2.1` The `__tag__:__client_ip__` field is a reserved field in Simple Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields. Important If a log is processed by using the data transformation feature or a Logtail plug-in, the key in a tag field is converted to a common key. If you want to search for the log, you must enclose the name of the tag field in double quotation marks ("") in the search statement. Example: `"__tag__:__client_ip__":192.0.2.1`.
Logs that contain `192.168.XX.XX`	`* \| select * from log where key like '192.168.%.%'` For more information, see Use the LIKE clause to implement fuzzy match.
Logs in which the remote_user field is not empty	`not remote_user:""`
Logs in which the remote_user field is empty	`remote_user:""`
Logs in which the value of the remote_user field is not null	`not remote_user:"null"`
Logs that do not contain the remote_user field	`not remote_user:*`
Logs that contain the remote_user field	`remote_user:*`
Logs in which the value of the city field is not Shanghai	`not city:Shanghai` Note If you want to query Chinese strings, you must turn on Include Chinese when you configure indexes. For more information, see Create indexes.

Advanced search examples

Fuzzy search

Expected search result	Search statement
Logs that contain specific strings. The strings start with cn.	`cn*`
Logs in which the value of the region field starts with cn.	`region:cn*`
Logs in which the value of the region field contains cn*.	`region:"cn"` Note* In this search statement, `cn` is a complete string. Examples: If the content of a log is `region:cn,en` and the delimiter is a comma (,), Simple Log Service splits the log content into `region`, `cn`, and `en`. You can use the search statement to search for the log. If the content of a log is `region:cnhangzhou`, Simple Log Service considers `cn*hangzhou` as a whole. Therefore, you cannot use the search statement to search for the log.
Logs that contain specific strings. The strings start with mozi, end with la, and include one character between mozi and la.	`mozi?la`
Logs that contain specific strings. The strings start with mo, end with la, and include zero, one, or more characters between mo and la.	`mo*la`
Logs that contain specific strings. The strings start with moz or sa.	`moz* and sa*`
Logs in which the value of the region field ends with hai.	You cannot use a search statement to search for the logs. You can use the LIKE clause in an SQL statement to search for the logs. For more information, see Use the LIKE clause to implement fuzzy match. `* \| select * from log where region like '%hai'`

Delimiter-based searches

Simple Log Service splits the content of a log into multiple strings based on the delimiters that you specify. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If you leave the Delimiter parameter empty, Simple Log Service considers the value of each field as a whole. In this case, you can search for a log only by using a complete string or a fuzzy search. For more information about how to specify delimiters, see Create indexes.

For example, the value of the http_user_agent field is Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/192.0.2.0 Safari/537.2.

If you leave the Delimiter parameter empty, Simple Log Service considers the field value as a whole. In this case, you cannot search for logs by using the http_user_agent:Chrome search statement.
If you set the Delimiter parameter to , '";=()[]{}?@&<>/:\n\t\r, Simple Log Service splits the field value into Mozilla, 5.0, Windows, NT, 6.1, AppleWebKit, 537.2, KHTML, like, Gecko, Chrome, 192.0.2.0, Safari, and 537.2. In this case, you can search for logs by using the http_user_agent:Chrome search statement.

Note

If a search keyword contains delimiters, you can perform a phrase search or use a LIKE clause. Examples:

Phrase search: #"redo_index/1". For more information, see Phrase search.
LIKE clause: * | select * from log where key like 'redo_index/1'.

Expected search result	Search statement
Logs in which the value of the http_user_agent field contains Chrome	`http_user_agent:Chrome`
Logs in which the value of the http_user_agent field contains Linux and Chrome	`http_user_agent:"Linux Chrome"` `http_user_agent:Linux and http_user_agent:Chrome`
Logs in which the value of the http_user_agent field contains Firefox or Chrome	`http_user_agent:Firefox or http_user_agent:Chrome`
Logs in which the value of the request_uri field contains /request/path-2	`request_uri:/request/path-2`
Logs in which the value of the request_uri field starts with /request and does not contain /file-0	`request_uri:/request* not request_uri:/file-0`
Logs in which the phrase `redo_index/1` is completely matched	`#"redo_index/1"` `* \| select * from log where key like 'redo_index/1'` Note You can perform a phrase search or use a LIKE clause to completely match the phrase. If you perform an exact search, strings such as `redo_index` and `1` are matched.

Searches for the JSON logs whose field values are JSON objects or JSON arrays
If the value of a field is in the JSON format, you can set the data type of the field to JSON or use JSON functions to query and analyze the field. For more information, see FAQ about the query and analysis of JSON logs.