Simple Log Service provides the scan-based query feature that scans logs based on specified fields to return query results. This feature allows you to query logs without the need to configure indexes for the logs. This topic describes the scan-based query feature.
Background information
Simple Log Service provides the index-based query feature. However, in some scenarios, indexes cannot be created or used. Examples:
Indexes are not created for some fields to reduce costs. In this scenario, an urgent need to query historical logs may not be filled.
The length of a field value exceeds the upper limit supported by indexes. In this scenario, the excess part cannot be queried.
A large number of field names are involved and cannot be obtained in advance. In this scenario, field indexes cannot be created.
The type of a field is changed. In this scenario, the field cannot be queried.
To address the preceding issues, Simple Log Service provides the scan-based query and scan-based analysis features. You can use the features to query and analyze logs without the need to create indexes for the fields that are used for scanning. For more information about the scan-based analysis feature, see Scan-based analysis overview.
The scan-based query feature is upgraded to support the Simple Log Service Processing Language (SPL). For more information, see SPL overview. You can use SPL statements to extract structured information from the raw data that is obtained, process fields in the raw data, and filter the raw data. You can use multi-level pipeline cascading to export data that is processed by using the SPL syntax. For more information about the regions in which SPL is supported, see Supported regions.
Basic syntax
Syntax
Search statement | WHERE bool_expressionExample
status:200 | WHERE userId = '123'
How it works
When Simple Log Service receives a scan-based query request, Simple Log Service performs the following steps:
Executes the search statement to query logs.
ImportantThe search statement requires indexes. For example, before the
status:200 | WHERE userId = '123'statement is executed, you must create an index for the status field. You do not need to create an index for the userId field.Executes the WHERE clause to identify logs that meet the scan condition among the query results of the search statement and then returns the final results.
Comparison between index-based query and scan-based query
Item | Index-based query | Scan-based query |
Syntax |
|
|
Whether indexes need to be configured | Yes. | No. Important The search statement before the vertical bar (|) requires indexes. |
Support for an analytic statement | Supported. | Supported. |
Support for random page turning | Supported. | Not supported. Only continuous page turning is supported. You can move forward or backward only one page. |
Log histogram | The log histogram is displayed based on the results of the search statement. | The log histogram is displayed based on the results of the search statement and the scan progress. |
Operators and functions | Logical calculation and mathematical calculation are supported. Fuzzy search is supported. SQL functions are not supported. | Keywords such as LIKE, IN, BETWEEN, IS NULL, IS NOT NULL, and EXISTS are supported. Some SQL functions are supported. For more information, see Supported functions. |
Field types | Field types are determined by the data types that are specified in index configurations. For more information, see Data types. | The system considers the types of fields in the WHERE clause as text regardless of whether indexes are configured for the fields. Note If you want to use a function whose input fields are not of the text type, you must use the cast function to convert the types of the fields. An example of the function is abs. For more information, see cast function. |
Result size | The number of logs to return can be specified in the Simple Log Service console or by calling an SDK. The maximum number is 100. | If one of the following conditions is met, the system stops the current scan and returns results:
|
Fees | You are charged for index traffic and index storage. For more information, see Billable items of pay-by-feature. | You are charged for scans based on the scan traffic, which is equivalent to the amount of data returned after scanning. The system identifies logs based on the results of index-based query. |
Benefits
Low costs: You do not need to create indexes. You are not charged for index traffic or index storage.
Flexible implementation: You can specify field types based on your business requirements regardless of whether indexes are created or index types are specified.
Optimized search capability: More functions are supported.
Limits
Random page turning is not supported.
Supported functions
Mathematical calculation functions
Function | Syntax | Description |
abs(x) | Calculates the absolute value of x. | |
cbrt(x) | Calculates the cube root of x. | |
ceil(x) | Rounds x up to the nearest integer. The ceil function is an alias of the ceiling function. | |
ceiling(x) | Rounds x up to the nearest integer. | |
e() | Returns the value of e, which is the base of the natural logarithm. | |
exp(x) | Raises e to the power of x. | |
floor(x) | Rounds x down to the nearest integer. | |
from_base(x, y) | Converts x to a base-y number. | |
ln(x) | Calculates the natural logarithm of x. | |
log2(x) | Calculates the base-2 logarithm of x. | |
log10(x) | Calculates the base-10 logarithm of x. | |
mod(x, y) | Calculates the remainder of x divided by y. | |
pi() | Returns the value of π to 15 decimal places. | |
pow(x, y) | Raises x to the power of y. The pow function is an alias of the power function. | |
power(x, y) | Raises x to the power of y. | |
radians(x) | Converts an angle in degrees to its equivalent in radians. | |
rand() | Returns a random number. | |
random() | Returns a random number in the range [0,1). | |
random(x) | Returns a random number in the range [0,x). | |
round(x) | Rounds x to the nearest integer. | |
round(x, n) | Rounds x to the nearest decimal with n decimal places. | |
sqrt(x) | Calculates the square root of x. | |
to_base(x, y) | Converts x to a base-y string. | |
truncate(x) | Removes the fractional part of x. | |
width_bucket(x, bound1, bound2, numBuckets) | Divides a numeric range into buckets of equal width and returns the bucket number of x. | |
width_bucket(x, bins) | Returns the bucket number of x in the range of buckets that are specified by an array. |
String functions
Function | Syntax | Description |
chr(x) | Converts an ASCII code to characters. | |
codepoint(x) | Converts characters to an ASCII code. | |
concat(x, y...) | Concatenates multiple strings into one string. | |
length(x) | Returns the length of a string. | |
lower(x) | Converts the characters in a string to lowercase letters. | |
lpad(x, length, lpad_string) | Left pads a string to a specified length by using a specified character and returns the result string. | |
ltrim(x) | Removes spaces from the start of a string. | |
replace(x, sub_string ) | Removes the matched characters from a string. | |
replace(x, sub_string, replace_string) | Replaces the matched characters in a string with specified characters. | |
reverse(x) | Returns a string in reverse order. | |
rpad(x, length, rpad_string) | Right pads a string to a specified length by using a specified character and returns the result string. | |
rtrim(x) | Removes spaces from the end of a string. | |
split(x, delimeter) | Splits a string by using a specified delimiter and returns a set of substrings. | |
split(x, delimeter, limit) | Splits a string by using a specified delimiter and returns a set of substrings. The number of substrings that can be generated is specified by limit. | |
split_part(x, delimeter, part) | Splits a string by using a specified delimiter and returns the substring at a specified position. | |
strpos(x, sub_string) | Returns the position of a specified substring in a string. | |
strpos(x, sub_string, instance) | Returns the position of the nth occurrence of a specified substring in a string. n is specified by instance, which must be a positive integer. The positions of substrings start from 1. If the specified substring is not found, the function returns 0. | |
substr(x, start) | Returns the substring at a specified position in a string. | |
substr(x, start, length) | Returns the substring at a specified position in a string. The length of the substring is specified. | |
to_utf8(x) | Converts a string to a UTF-8 encoded string. | |
trim(x) | Removes spaces from the start and end of a string. | |
upper(x) | Converts the characters in a string to uppercase letters. |
Date and time functions
Category | Function | Syntax | Description |
Date and time functions | date_format(x, format) | Converts a datetime expression that can return a timestamp value to a datetime expression in a specified format. | |
from_unixtime(x) | Converts a UNIX timestamp to a datetime expression that can return a timestamp value and does not contain a time zone. | ||
from_unixtime(x, time zone) | Converts a UNIX timestamp to a datetime expression that can return a timestamp value and contains a time zone. | ||
from_unixtime(x, hours, minutes) | Converts a UNIX timestamp to a datetime expression that can return a timestamp value and contains a time zone. In the datetime expression, the values for hours and minutes indicate the offset of the time zone. | ||
to_unixtime(x) | Converts a datetime expression that can return a timestamp value to a UNIX timestamp. | ||
parse_datetime function | parse_datetime(x, format) | Converts a string to a datetime expression that can return a timestamp value and contains a time zone based on the setting of format. | |
Date and time extraction functions | day(x) | Returns the day of the month from a datetime expression. The day function is equivalent to the day_of_month function. | |
day_of_month(x) | Returns the day of the month from a datetime expression. The day_of_month function is equivalent to the day function. | ||
day_of_week(x) | Returns the day of the week from a datetime expression. The day_of_week function is equivalent to the dow function. | ||
day_of_year(x) | Returns the day of the year from a datetime expression. The day_of_year function is equivalent to the doy function. | ||
dow(x) | Returns the day of the week from a datetime expression. The dow function is equivalent to the day_of_week function. | ||
doy(x) | Returns the day of the year from a datetime expression. The doy function is equivalent to the day_of_year function. | ||
extract(field from x) | Returns the specified field from a datetime expression. The field can be a date or time. | ||
hour(x) | Returns the hour of the day from a datetime expression. The 24-hour clock is used. | ||
minute(x) | Returns the minute of the hour from a datetime expression. | ||
millisecond function | millisecond(x) | Returns the millisecond of the second from a datetime expression. | |
month(x) | Returns the month of the year from a datetime expression. | ||
quarter(x) | Returns the quarter of the year on which a specified date falls. | ||
second(x) | Returns the second of the minute from a datetime expression. | ||
year(x) | Returns the year of a specified date. | ||
year_of_week(x) | Returns the year on which a specified date falls in the ISO week date system. The year_of_week function is equivalent to the yow function. | ||
yow(x) | Returns the year on which a specified date falls in the ISO week date system. The yow function is equivalent to the year_of_week function. | ||
Time interval functions | date_trunc(unit, x) | Truncates a datetime expression based on the time unit that you specify. The expression can be truncated based on the millisecond, second, minute, hour, day, month, or year. | |
date_add(unit, N, x) | Adds N to the value of the x field based on the unit that you specify. | ||
date_diff(unit, x, y) | Returns the difference between two time expressions. For example, you can calculate the difference between x and y based on a specified time unit. |
Regular expression functions
Function | Syntax | Description |
regexp_extract_all(x, regular expression) | Extracts the substrings that match a specified regular expression from a specified string and returns an array of all matched substrings. | |
regexp_extract_all(x, regular expression, n) | Extracts the substrings that match a specified regular expression from a specified string and returns an array of substrings that match the nth capturing group in the regular expression. | |
regexp_extract(x, regular expression) | Extracts the first substring that matches a specified regular expression from a specified string and returns the substring. | |
regexp_extract(x, regular expression, n) | Extracts the substrings that match a specified regular expression from a specified string and returns the first substring that matches the nth capturing group in the regular expression. | |
regexp_like(x, regular expression) | Checks whether a specified string matches a specified regular expression. | |
regexp_replace(x, regular expression) | Deletes the substrings that match a specified regular expression from a specified string and returns the substrings that remain. | |
regexp_replace(x, regular expression, replace string) | Replaces the substrings that match a specified regular expression in a specified string and returns the result string. |
JSON functions
Function | Syntax | Description |
json_extract_scalar(x, json_path) | Returns a set of scalar values from a JSON object or a JSON array. The scalar values can be of the string, integer, or Boolean type. |