Search syntax and examples - Simple Log Service - Alibaba Cloud Documentation Center

Simple Log Service provides a set of search syntax that can be used to specify query conditions. Logs that meet the specified query conditions are returned. You can use the query results in analytic statements for complex data analysis and processing.

Search syntax structure

Syntax

Each query statement consists of a search statement and an analytic statement. The search statement and the analytic statement are separated by a vertical bar (|). A search statement can be executed alone. However, an analytic statement must be executed together with a search statement. You can use the log analysis feature to analyze the data in search results. You can also use the feature to analyze all data in a Logstore.

Important

You do not need to specify the FROM or WHERE clause in an analytic statement. By default, all data of the current Logstore is analyzed.
You do not need to append a semicolon (;) to an analytic statement as the statement terminator.
Analytic statements are not case-sensitive.

Syntax

Search statement|Analytic statement

Statement

Description

Search statement

A search statement specifies one or more search conditions. A condition can be a keyword, a numeric value, a numeric value range, a space character, or an asterisk (*).

If you specify a space character or an asterisk (*) as the search statement, no conditions are specified and all logs are returned. For more information, see Search syntax.

Analytic statement

An analytic statement is used to aggregate or analyze search results or all data in a Logstore.

Example

* | SELECT status, count(*) AS PV GROUP BY status

Statement writing process

You can write a search statement in the following procedure: determine search syntax based on the search type, specify an operator based on the data types of fields, and then determine a match mode.

Step 1: Select a search type

Important

If you want to analyze a specific field, you must create field indexes and use a SELECT statement. For more information about how to create field indexes and full-text indexes, see Create indexes.
Query and analysis results vary based on index configurations. If you create full-text indexes and field indexes, the field indexes take precedence.

If you create only full-text indexes for your Logstore, you can use only full-text search syntax to specify query conditions.
If you create only field indexes for your Logstore, the search syntax that you can use vary based on the data types of fields.
- Double and long: You can use only field-specific search syntax to specify query conditions.
- Text: If you can identify the field with which a keyword is associated and field indexes are created for the field, we recommend that you use field-specific search syntax to specify query conditions. If you cannot identify the field with which the keyword is associated, we recommend that you use full-text search syntax.
  - If you use full-text search syntax but full-text indexes are not created for your Logstore, you can query data only from fields whose indexed data is of the text type.
  - If you use full-text search syntax and full-text indexes are created for your Logstore, you can query data from all fields. In this case, the indexed data of all fields is of the text type.

Full-text search syntax

You cannot query data from a specific field based on full-text search syntax. Full-text search syntax:

keywords1 [ [ and | or | not ] keywords2 ] ...

keywords1 specifies the keyword that you want to use to query data. You can also use asterisks (*) and question marks (?) to perform fuzzy match. To combine query conditions, you can use operators such as and and or.

Example 1: Query logs that contain the Nginx keyword. Search statement: Nginx.

Example 2: Query logs that contain the Nginx or Tomcat keyword. Search statement: Nginx or Tomcat.

Field-specific search syntax

You can query data from specific fields based on field-specific search syntax. The fields can be of the text, long, double, or JSON type. Field-specific search syntax:

indexname1 [ : | > | >= | < | <= | = | in ] keyword1 [ [ and | or | not ] indexname2 ... ]

Note

indexname1 specifies the name of the field from which you want to query data. If a proper noun such as a field name or a table name contains special characters such as spaces and Chinese characters or syntax keywords such as and and or, you must enclose the characters or keywords in double quotation marks (""). For more information, see How do I use quotation marks in query statements?
If indexes are created for fields of the long or double type, you can use the following comparison operators: >, >=, <, <=, =, and in.

Example 1: Query logs whose requestMethod field value is GET. Search statement: requestMethod: GET.

Example 2: Query logs whose requestTimeSpend field value is greater than 500 ms. Search statement: requestTimeSpend > 500. The data type of the field is long.

Example 3: Query logs whose requestMethod field value is GET and requestTimeSpend field value is greater than 500 ms. Search statement: requestMethod: GET and requestTimeSpend > 500.

Step 2: Determine a field data type

Field data types

When you write a search statement, take note of the characteristics of field data types and use the required operators. This way, you can use the statement to obtain logs in an efficient and accurate manner.

Note

By default, if you enable full-text indexing, the data types of all fields in a log except the __time__ field are set to text.
If you enable field indexing, the data types of the related fields can be Long or Double, Text, and JSON.

Field data type	Description	Supported operator
Text	You can query data of the string type by setting the data types of related fields to text. By default, if you enable full-text indexing, the data types of all fields in a log except the `__time__` field are set to text.	`and`, `or`, `not`, `()`, `:`, `""`, `\`, `*`, and `?`.
Long or double	You can query the value of a field by using a numeric range only if you set the data type of the field to long or double. If you do not set the data type of a field to double or long or the syntax of the numeric range is invalid, Simple Log Service performs a full-text search, and the search result may be different from the expected result. For example, if you execute the `owner_id>100` search statement and the data type of the `owner_id` field is not double or long, logs that contain `owner_id`, `>`, and `100` are returned. In this example, the greater-than sign (>) is not a delimiter. If you change the data type of a field from text to double or long, you can use only the equal-to sign (=) to query data. If you want to use ranges and comparison operators such as the greater-than (>) and less-than (<) signs to query data, you must reindex the data. For more information, see Reindex logs for a Logstore.	`and`, `or`, `not`, `()`, `>`. `>=`, `<`, `<=`, `=`, and `in`.
JSON	You can set the data type of a field in JSON objects to long, double, or text based on the field value, and turn on Enable Analytics for the field.	Specify operators based on the data types of the fields in JSON objects.

Operators

Important

The letters of the in operator must be in lowercase. Other operators are not case-sensitive.
Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use these operators as keywords, you must enclose the operators in double quotation marks ("").
The following operators are listed in descending order of priority:
1. Colons (:)
2. Double quotation marks ("")
3. Parentheses ()
4. and and not
5. or

Operator	Description
:	This operator is used for field-specific searches based on the key:value format. Example: `request_method:GET`. If a field name or a field value contains special characters such as spaces, colons (:), and hyphens (-), you must enclose the field name or field value in double quotation marks (""). Example: `"file info":apsara`.
and	The and operator. Example: `request_method:GET and status:200`. If no syntax keywords exist among multiple search keywords, the search keywords are evaluated by using the and operator. For example, `GET 200 cn-shanghai` is equivalent to `GET and 200 and cn-shanghai`.
or	The or operator. Example: `request_method:GET or status:200`.
not	The not operator. Example: `request_method:GET not status:200` or `not status:200`.
( )	This operator is used to increase the priority of the query conditions that are enclosed in parentheses (). Example: `(request_method:GET or request_method:POST) and status:200`.
""	This operator is used to enclose a syntax keyword. If a syntax keyword is enclosed in double quotation marks (""), the keyword is converted to an ordinary word. For example, `"and"` returns the logs that contain the word and. In this case, and is not an operator. In a field-specific search, the words that are enclosed in double quotation marks ("") are considered as a whole.
\	The escape character. This character is used to escape double quotation marks (""). Double quotation marks ("") can indicate themselves only after they are escaped. For example, if the content of a log is `instance_id:nginx"01"`, you can execute the `instance_id:nginx\"01\"` statement to search for the log.
*	The wildcard character. This character is used to match zero, one, or multiple characters. Example: `host:aliyundc`. Note* Simple Log Service searches all logs and obtains up to 100 words that match the specified conditions. Then, Simple Log Service returns the logs that contain one or more of these words and match the query conditions.
?	The wildcard character. This character is used to match a single character. Example: `host:aliyund?c`.
>	This operator is used to query the logs in which the value of a field is greater than a specified numeric value. Example: `request_time>100`.
>=	This operator is used to query the logs in which the value of a field is greater than or equal to a specified numeric value. Example: `request_time>=100`.
<	This operator is used to query the logs in which the value of a field is less than a specified numeric value. Example: `request_time<100`.
<=	This operator is used to query the logs in which the value of a field is less than or equal to a specified numeric value. Example: `request_time<=100`.
=	This operator is used to query the logs in which the value of a field is equal to a specified numeric value. Equal-signs (=) and colons (:) have the same effect on fields of the double or long type. For example, `request_time=100` is equivalent to `request_time:100`.
in	This operator is used to query the logs in which the value of a field is within a specified numeric range. Brackets [] indicate a closed interval, and parentheses () indicate an open interval. A space is used to separate two numbers in a numeric range. Example: `request_time in [100 200]` or `request_time in (100 200]`. Important The letters of the in operator must be in lowercase.
__source__	This operator is used to query the logs of a specified log source. Wildcard characters are supported. Example: `__source__:192.0.2.`. Important* The __source__ field is a reserved field in Simple Log Service. This field can be abbreviated to source. If you configure a custom source field, the custom field conflicts with the reserved source field in Simple Log Service. If you want to search for the custom field, you must use Source or SOURCE in the search statement.
__tag__	This operator is used to query logs by using metadata. Example: `__tag__:__receive_time__:1609837139`.
__topic__	This operator is used to query the logs of a specified log topic. Example: `__topic__:nginx_access_log`.

Step 3: Determine a match mode

You can use exact searches or fuzzy searches based on keywords and your business requirements. If the required logs are not returned, troubleshoot the issue based on the instructions in What do I do if no results are returned when I query a log?

Search type

Description

Example

Exact search

Complete words are used for searches.

Simple Log Service uses word segmentation to query logs. In an exact search, phrases cannot be completely matched. For example, the abc def search statement returns the logs that contain abc and def. The phrase abc def cannot be completely matched. If you want the abc def phrase to be completely matched, you can perform a phrase search or use a LIKE clause. For more information, see Phrase search and How do I query logs by using exact match?

host:example.com: returns the logs in which the value of the host field contains example.com.
PUT and cn-shanghai: returns the logs that contain the PUT and cn-shanghai keywords.
* | Select * where http_user_agent like '%like Gecko%': returns the logs in which the value of the http_user_agent field contains the like Gecko phrase.
#"redo_index/1": returns the logs that contain the redo_index/1 phrase.

Fuzzy search

You can add an asterisk (*) or a question mark (?) as a wildcard character to the middle or end of a word in a search statement when you perform a fuzzy search. The word must be 1 to 64 characters in length. If a word contains a wildcard character, Simple Log Service searches all logs and obtains up to 100 words that match the word. Then, Simple Log Service returns the logs that contain one or more of these words. If you specify more accurate words, the search result is more accurate.

Important

You cannot add an asterisk (*) or a question mark (?) to the start of a word.
The long and double data types do not support asterisks (*) or question marks (?) in fuzzy searches. You can specify a numeric range to perform a fuzzy search. Example: status in [200 299].

A fuzzy search is performed based on samples by using the following mechanism:

If you enable the field indexing feature and specify a field to query logs, Simple Log Service obtains random samples from the indexed data of the field and returns results. Simple Log Service does not perform full-text scans.
If you enable the full-text indexing feature and do not specify a field to query logs, Simple Log Service obtains random samples from the indexed data of all fields and returns results. Simple Log Service does not perform full-text scans.

request_time>60 and request_method:Ge*: returns the logs in which the value of the request_time field is greater than 60 and the value of the request_method field starts with Ge.
addr*: obtains 100 words that start with addr from all logs and returns the logs that contain one or more of these words.
host:www.yl*: obtains 100 words that start with www.yl from the values of the host field in all logs and returns the logs that contain one or more of these words.

For more information, see How do I query logs by using fuzzy match?

Examples of search statements

If you execute a search statement on different logs based on different index configurations, the statement returns different results. The examples provided in this section are based on the following sample log and index configurations.

Sample log

An NGINX access log is used as the sample log.

日志样例

Index configurations

Before you can execute a search statement, make sure that indexes are configured. For more information, see Create indexes.

On the query and analysis page of your Logstore, click Index Attributes in the upper-right corner and select Attributes.

Check whether indexes are configured. The following figure shows fields for which indexes are configured.

Common search examples

Expected search result	Search statement	Debugging
Logs that record successful GET requests (status codes: 200 to 299)	`request_method:GET and status in [200 299]`	Debugging
Logs that record GET requests and in which the source region of the requests is not the China (Hangzhou) region	`request_method:GET not region:cn-hangzhou`	None
Logs that record GET requests or POST requests	`request_method:GET or request_method:POST`	Debugging
Logs that do not record GET requests	`not request_method:GET`	Debugging
Logs that record successful GET requests or successful POST requests	`(request_method:GET or request_method:POST) and status in [200 299]`	Debugging
Logs that record failed GET requests or failed POST requests	`(request_method:GET or request_method:POST) not status in [200 299]`	Debugging
Logs that record successful GET requests (status codes: 200 to 299) and in which the request duration is less than 60 seconds	`request_method:GET and status in [200 299] not request_time>=60`	Debugging
Logs in which the request duration is equal to 60 seconds	`request_time:60`	Debugging
Logs in which the request duration is equal to 60 seconds	`request_time=60`	Debugging
Logs in which the request duration is greater than or equal to 60 seconds and is less than 200 seconds	`request_time>=60 and request_time<200`	Debugging
	`request_time in [60 200)`	Debugging
Logs in which the request_time field is empty or the value of the field is an invalid number	`request_time:* not request_time > -10000000000` Note `request_time:` is used to check whether the `request_time` field exists. In this search statement, the asterisk () does not indicate a fuzzy search.	Debugging
Logs that contain the request_time field and in which the value of the field is a number	`request_time > -1000000000`	Debugging
Logs that contain and	`"and"` Note In this search statement, and is a common string but not an operator.	Debugging
Logs in which the value of the request method field is PUT	`"request method":PUT` Important The name of the request method field contains spaces. You must enclose the field name in double quotation marks ("") in a search statement.	None
Logs whose topic is HTTPS or HTTP	`__topic__:HTTPS or __topic__:HTTP`	None
Logs that are collected from the 192.0.2.1 host	`__tag__:__client_ip__:192.0.2.1` The `__tag__:__client_ip__` field is a reserved field in Simple Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields. Important If a log is processed by using the data transformation feature or a Logtail plug-in, the key in the __tag__:__client_ip__ field is converted to a common key. If you want to search for the log, you must enclose the name of the __tag__:__client_ip__ field in double quotation marks ("") in the search statement. Example: `"__tag__:__client_ip__":192.0.2.1`.	None
Logs whose IP addresses match `192.168.XX.XX`	`* \| select * from log where key like '192.168.%.%'` For more information, see Use the LIKE clause to implement fuzzy match.	None
Logs in which the remote_user field is not empty	`not remote_user:""`	Debugging
Logs in which the remote_user field is empty	`remote_user:""`	Debugging
Logs in which the value of the remote_user field is not null	`not remote_user:"null"`	Debugging
Logs that do not contain the remote_user field	`not remote_user:*`	Debugging
Logs that contain the remote_user field	`remote_user:*`	Debugging
Logs in which the value of the city field is not Shanghai	`not city:Shanghai` Note If you want to query Chinese strings, you must turn on Include Chinese when you configure indexes. For more information, see Create indexes.	None

Advanced search examples

Fuzzy searches

Expected search result	Search statement	Debugging
Logs that contain specific words. The words start with cn.	`cn*`	Debugging
Logs in which the value of the region field starts with cn.	`region:cn*`	None
Logs in which the value of the region field contains cn*.	`region:"cn"` Note* In this search statement, `cn` is a complete string. Examples: If the content of a log is `region:cn,en` and the delimiter is a comma (,), Simple Log Service splits the log content into `region`, `cn`, and `en`. You can use the search statement to search for the log. If the content of a log is `region:cnhangzhou`, Simple Log Service considers `cn*hangzhou` as a whole. In this case, you cannot use the search statement to search for the log.	None
Logs that contain specific words. The words start with mozi, end with la, and include one character between mozi and la.	`mozi?la`	Debugging
Logs that contain specific words. The words start with mo, end with la, and include zero, one, or more characters between mo and la.	`mo*la`	Debugging
Logs that contain specific words. The words start with moz or sa.	`moz* and sa*`	Debugging
Logs in which the value of the region field ends with hai.	You cannot use a search statement to search for the logs. You can use the LIKE clause in an SQL statement to search for the logs. For more information, see Use the LIKE clause to implement fuzzy match. `* \| select * from log where region like '%hai'`	None

Delimiter-based searches

Simple Log Service splits the content of a log into multiple words based on the delimiters that you specify. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If you leave Delimiter empty, Simple Log Service considers the value of each field as a whole. In this case, you can search for a log only by using a complete string or performing a fuzzy search. For more information about how to specify delimiters, see Create indexes.

For example, the value of the http_user_agent field is Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/192.0.2.0 Safari/537.2.

If you leave Delimiter empty, Simple Log Service considers the field value as a whole. In this case, you cannot search for logs by using the http_user_agent:Chrome search statement.
If you set Delimiter to , '";=()[]{}?@&<>/:\n\t\r, Simple Log Service splits the field value into Mozilla, 5.0, Windows, NT, 6.1, AppleWebKit, 537.2, KHTML, like, Gecko, Chrome, 192.0.2.0, Safari, and 537.2. In this case, you can search for logs by using the http_user_agent:Chrome search statement.

Note

If a search keyword contains delimiters, you can perform a phrase search or use a LIKE clause. Examples:

Phrase search: #"redo_index/1". For more information, see Phrase search.
LIKE clause: * | select * from log where key like 'redo_index/1'.

Expected search result	Search statement	Debugging
Logs in which the value of the http_user_agent field contains Chrome	`http_user_agent:Chrome`	Debugging
Logs in which the value of the http_user_agent field contains Linux and Chrome	`http_user_agent:Linux and http_user_agent:Chrome`	Debugging
	`http_user_agent:"Linux Chrome"`	Debugging
Logs in which the value of the http_user_agent field contains Firefox or Chrome	`http_user_agent:Firefox or http_user_agent:Chrome`	Debugging
Logs in which the value of the request_uri field contains /request/path-2	`request_uri:/request/path-2`	Debugging
Logs in which the value of the request_uri field starts with /request and does not contain /file-0	`request_uri:/request* not request_uri:/file-0`	Debugging
Logs in which the `redo_index/1` phrase is completely matched	`#"redo_index/1"` `* \| select * from log where key like 'redo_index/1'` Note You can perform a phrase search or use a LIKE clause to completely match the phrase. If you perform an exact search, words such as `redo_index` and `1` are matched.	None

Query examples in special scenarios

Search statements

If a field name or a field value contains special characters such as spaces, Chinese characters, colons (:), and hyphens (-) or contains syntax keywords such as and and or, you must enclose the field name or field value in double quotation marks ("").
Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use the operators as keywords, you must enclose the operators in double quotation marks ("").
If a log is processed by using the data transformation feature or a Logtail plug-in, the key in a tag field is converted to a common key. If you want to search for the log, you must enclose the name of the key in double quotation marks (""). Example: "__tag__:__client_ip__":192.0.2.1. The __tag__:__client_ip__ field is a reserved field of Simple Log Service. The field indicates the IP address of the host from which the log is collected. For more information, see Reserved fields.

Expected query result	Search statement	Debugging
Logs in which the value of the `request method` field contains `PUT`. The name of the `request method` field contains a space.	`"request method":PUT`	None
Logs that are collected from the 192.0.2.1 host.	`"__tag__:__client_ip__":192.0.2.1`	None

Analytic statements

If a proper noun such as a field name or a table name contains special characters such as spaces, Chinese characters, colons (:), and hyphens (-) or contains syntax keywords such as and and or, you must enclose the proper noun in double quotation marks ("") in analytic statements.
If specific characters represent a string, you must use single quotation marks ('') to enclose the characters in analytic statements. For example, 'status' indicates the status string, and status or "status" indicates the status log field.

Expected query result	Query statement	Debugging
Logs whose IP addresses match `192.168.XX.XX`	`* \| select * from log where key like '192.168.%.%'`	None

For more information, see How do I use quotation marks in query statements?

References

How do I use quotation marks in query statements?
For more information about analytic functions and syntax, see Analytic functions and syntax.
Optimize query performance
Troubleshooting
- FAQ about log query
- How do I resolve common errors that may occur when I query and analyze logs?
Log query examples
- How do I precisely sort query logs by time?
- Query and analyze website logs
- Analyze Layer 7 access logs of SLB
- You can query JSON logs whose field values are JSON objects or JSON arrays. For more information about query and analysis examples, see Query and analyze JSON logs and FAQ about the query and analysis of JSON logs.