Consume data based on rules in Simple Log Service - Simple Log Service

Simple Log Service provides the rule-based consumption feature to consume logs that meet specific conditions in real time. This topic describes the rule-based consumption feature.

Important

The rule-based consumption feature is available in the following regions: China (Beijing), China (Chengdu), China (Guangzhou), Malaysia (Kuala Lumpur), Philippines (Manila), UK (London), and US (Silicon Valley).

Scenarios

Consume data over the Internet with reduced traffic costs
For example, you want to consume logs over the Internet after you upload the logs to Simple Log Service. Then, you want to filter the logs and distribute the logs to the internal system. In this case, you can use the rule-based consumption feature to filter logs in Simple Log Service. This way, a large number of invalid logs are not shipped to consumers. This helps reduce traffic costs.
Compute data in an on-premises machine with reduced computing resources and increased computing speed
For example, you want to consume logs by using an on-premises machine after you upload the logs to Simple Log Service. Then, you want to compute data on the on-premises machine. In this case, you can use the rule-based consumption feature to compute data based on specific rules in Simple Log Service. This helps reduce a large number of computing resources and increase the computing speed.

Basic syntax

*| where bool_expression

Example:

*| where level = '123'

Limits

A query statement must be less than 10 KB in length.
A query statement must start with *| where. You can modify the WHERE clause.
You cannot use the __time__ field to filter logs.
You cannot specify a JOIN, ORDER BY, GROUP BY, HAVING, or AGGREGATE clause in a query statement.
Subqueries are not supported. You cannot specify a column as the result set.
All fields in a WHERE clause are of the text type. If you want to use a function whose input parameter is not of the text type, such as the abs function, you must use the cast function to convert the data type of the corresponding field. For more information, see Data type conversion functions.
In scenarios in which no data is returned, all data is filtered out in a request. In this case, the value of the nextCursor parameter in the returned result is changed.

How rule-based consumption works

Simple Log Service provides the rule-based consumption feature to match logs based on specific filter rules and ship the matched logs to consumers.

Query parser: parses and verifies query statements and then generates filter rules.
Data filter: scans all raw data to match the data that meets the filter rules.

Differences between common data consumption and rule-based consumption

Item	Common data consumption	Rule-based consumption
Data size	The maximum size of compressed data that can be read after you call the LogGroupList operation is 1 MB.	The maximum size of compressed data that can be read after you call the LogGroupList operation is 1 MB.
Operators	No	SQL subsets are supported. Data is processed by row. For more information, see Operators.
Consumption-related quota	If the number of concurrent data consumption jobs exceeds the limit after read and write traffic is throttled, error 403 is reported.	If the number of concurrent data consumption jobs exceeds the limit after read and write traffic is throttled, error 403 is reported.

Billing

The rule-based consumption feature is free of charge during the public preview. After the feature is officially released, you are charged fees when you use the feature. For more information, see the announcements and updates of Simple Log Service. During the public preview, you are charged for the read and write traffic that is generated when you consume data based on the amount of data that is filtered based on rules. For more information, see Billable items of pay-by-feature.

Operator

The following table describes the operators that are supported by the rule-based consumption feature.

Operator	Description
=	Matches logs in which the value of a field is equal to a value. Example: `*\| where request_method = 'GET'`.
!=	Matches logs in which the value of a field is not equal to a value. Example: `*\| where request_method != 'GET'`.
>	Matches logs in which the value of a field is greater than a value. Example: `*\| where cast(status as bigint) > 200`.
>=	Matches logs in which the value of a field is greater than or equal to a value. Example: `*\| where cast(status as bigint) >= 200`.
<	Matches logs in which the value of a field is less than a value. Example: `*\| where cast(status as bigint) < 200`.
<=	Matches logs in which the value of a field is less than or equal to a value. Example: `*\| where cast(status as bigint) <= 300`.
like	Matches a specific character pattern in a string and queries logs in which a field contains a value. Example: `*\| where http_user_agent like 'like Gecko%'`.
in	Specifies multiple values in a WHERE clause. Example: `*\| where request_method in ('GET', 'POST')`.
is null	Matches logs in which the value of a field is null. Example: `*\| where request_method is null`.
is not null	Matches logs in which the value of a field is not null. Example: `*\| where request_method is not null`.
and	The and operator. Example: `*\| where request_method = 'GET' and status = '200'`.
or	The or operator. Example: `*\| where request_method = 'GET' or status = '200'`.
not	The not operator. Example: `*\| where request_method = 'GET' not status = '200'`.
( )	This operator is used to increase the priority of the query conditions that are enclosed in parentheses (). Example: `*\| where (request_method = 'GET' or request_method = 'POST') and status = '200'`.

Supported functions

For more information, see Supported functions.

Operator	Description
=	Matches logs in which the value of a field is equal to a value. Example: `*\| where request_method = 'GET'`.
!=	Matches logs in which the value of a field is not equal to a value. Example: `*\| where request_method != 'GET'`.
>	Matches logs in which the value of a field is greater than a value. Example: `*\| where cast(status as bigint) > 200`.
>=	Matches logs in which the value of a field is greater than or equal to a value. Example: `*\| where cast(status as bigint) >= 200`.
<	Matches logs in which the value of a field is less than a value. Example: `*\| where cast(status as bigint) < 200`.
<=	Matches logs in which the value of a field is less than or equal to a value. Example: `*\| where cast(status as bigint) <= 300`.
like	Matches a specific character pattern in a string and queries logs in which a field contains a value. Example: `*\| where http_user_agent like 'like Gecko%'`.
in	Specifies multiple values in a WHERE clause. Example: `*\| where request_method in ('GET', 'POST')`.
is null	Matches logs in which the value of a field is null. Example: `*\| where request_method is null`.
is not null	Matches logs in which the value of a field is not null. Example: `*\| where request_method is not null`.
and	The and operator. Example: `*\| where request_method = 'GET' and status = '200'`.
or	The or operator. Example: `*\| where request_method = 'GET' or status = '200'`.
not	The not operator. Example: `*\| where request_method = 'GET' not status = '200'`.
( )	This operator is used to increase the priority of the query conditions that are enclosed in parentheses (). Example: `*\| where (request_method = 'GET' or request_method = 'POST') and status = '200'`.