Query strings are used in domain specific languages (DSLs) to filter log data and simplify data processing. This topic describes the query string syntax used for different functions.

Functions

The following table describes functions that use query strings.
Type Function Scenario
Event check functions e_search Query strings are used to check whether the value of a field in an event meets specific conditions.
Resource functions res_log_logstore_pull Query strings are used to configure a blacklist or whitelist to filter table data retrieved from a Logstore.
res_rds_mysql Query strings are used to configure a blacklist or whitelist to filter table data retrieved from an ApsaraDB RDS for MySQL database.
Event mapping functions e_search_table_mapande_search_dict_map Query strings are used to match key-value pairs in a dictionary.

Features

The following table lists the search features that are available for field search and full-text search.
Feature Field search Full-text search
Substring search Supported Supported
Wildcard characters *? search Supported Supported
Exact match Supported Unsupported
Search with regular expressions Supported Unsupported
Numeric range search Supported Unsupported
Numeric value comparison Supported Unsupported
Logical operators such as AND, OR, NOT, and their combinations Supported Supported

Escape special characters

Some special characters such as asterisks (*) and backslashes (\) must be escaped in query strings.

Escape special characters in a field name

A field name cannot be enclosed with double quotation marks ("). Special characters must be escaped by using backslashes (\). Examples :
  • \*\(\1+1\)\?: abc: The format of the field name in this example is valid. The field name is not enclosed with double quotation marks ("). Special characters are escaped by using backslashes (\).
  • "content": abc: The format of the field name in this example is invalid. The field name cannot be enclosed with double quotation marks (").
Escape special characters in a field value
  • The "and \ characters must be escaped by using backslashes (\),
    for example, content: "abc\"xy\\z".
    Note In the preceding example, the whole string can be enclosed with ' or ''. However, you can enclose the substring only with ''. We recommend that you enclose a string with ' and to enclose a substring with '', for example, e_search('field1:"string" ').
  • The * and ? characters contained in a query string must be escaped by using backslashes (\). Otherwise, the characters are considered wildcard characters.
  • A field value that contains only letters, digits, underscores (_), hyphens (-), asterisks (*), and question marks (?) does not need to be enclosed with double quotation marks ("). Examples:
    • status: "\*\?()[]:=": The field value in this example is valid. The value contains special characters and is enclosed with double quotation marks ("). The *, ?, and \ characters in the value are escaped.
    • content: ()[]:=: The field value in this example is invalid. The value must be enclosed with double quotation marks (").
    • status: active\*test: The field value in this example is valid. The value contains an asterisk (*) and does not contain other special characters. The asterisk (*) is escaped, and the value does not need to be enclosed with double quotation marks (").
    • status: active\? test: The field value in this example is valid. The value contains a question mark (?) and does not contain other special characters. The question mark (?) is escaped, and the value does not need to be enclosed with double quotation marks (").
Note Double quotation marks (") are used to enclose a field value. Single quotation marks (') are not supported. For example, e_search("domain: '/url/test.jsp'") is invalid while e_search('domain: "/url/test.jsp"') is valid.

Substring search

The substring search feature supports full-text search and field search.

Full-text search: searches for substrings in all fields.
  • Syntax
    e_search('substring')
  • Examples
    • e_search('"error"'): searches for a substring.
    • e_search('"active error"'): searches for a substring that contains a space character.
    • e_search('active error'): searches for multiple substrings. Multiple substrings are associated with one another through the OR logical operator.
Field search: searches for a substring in specific fields.
  • Syntax
    e_search('...')
  • Examples
    • e_search('status: active'): searches for a substring.
    • e_search('author: "john smith"'): searches for a substring that contains a space character.
    Note e_search('field: active error'): searches the specified field for the substring "active" or searches all logs for the substring "error". The query string in this example is equivalent to field:active OR "error".

Search with wildcard characters

* represents zero or more strings. ? represents a normal character or a wide character. Both full-text search and field search support wildcard characters.

Full-text search: searches for substrings in all fields.
  • Syntax
    e_search('substring')
  • Examples
    • e_search('active*test'): The asterisk (*) is used to match zero or more characters. The query string does not need to be enclosed with double quotation marks (") because it contains only an asterisk (*).
    • e_search('error*occurs'): The asterisk (*) is used to match zero or more characters. error occurs, error in network connection occurs, and other similar substrings can be matched.
    • e_search('active? good'): The question mark (? ) is used to match only one character. The query string contains only the question mark, and it does not need to be enclosed by double quotation marks (").
    • e_search('ac*tive? good'): The query string is used for exact match.
    • e_search('ac*tive?? go*od'): The query string is used for exact match. A query string used for exact match can contain both question marks (?) and asterisks (*).
Field search: searches for a substring in specific fields.
  • Syntax
    e_search('field name: substring')
  • Examples
    • e_search('status: active*test'): The asterisk (*) is used to match zero or more characters.
    • e_search('status: active? good'): The question mark (?) is used to match only one character.

Exact match

Exact match requires that the entire field value be matched.
  • Syntax
    e_search(r'field name~="regular expression"')
  • Examples
    • e_search('author== "john smith"'): The value of the author field must completely match john smith.
    • e_search('status== ac*tive? good'): The query string contains wildcard characters and is used for exact match.

Search with regular expressions

Regular expressions can better match occurrences than wildcard characters.
  • Syntax
    e_search(r'field name~="regular expression"')
    Note
    • We recommend that you use r instead of backslashes (\) to escape special characters in regular expressions because backslashes are used in regular expressions.
    • By default, a regular expression is used to match occurrences of a substring in a string. For exact match, you need to start the regular expression with ^ and end the expression with $.
  • Examples
    • e_search(r'status~= "\d+"'): The value of the status field contains numbers.
    • e_search(r'status~= "^\d+$"'): The value of the status field is a numeric value.

Numeric value comparison

You can search for field values by comparing field values with numeric values.
  • You can use >, >=, =, <, and <= to compare field values with numeric values.
    e_search('age >= 18')  #  >=18
    e_search('age > 18')   #  > 18
    e_search('age = 18')   #  = 18
    e_search('age <= 18')  #  <=18
    e_search('age < 18')   #  < 18
  • You can also search for field values that are within a closed interval. You can use asterisks (*) in the interval to represent positive or negative infinity.
    e_search('count: [100, 200]') # >=100 and  <=200
    e_search('count: [*, 200]')   # <=200
    e_search('count: [200, *]')   # >=200

Logical relationships

Logical operators can be used between multiple search conditions. Parentheses () is used to nest search conditions.
Logical relationships Keyword
AND and, AND, and &&. The keyword is case insensitive.
OR or and OR. The keyword is case insensitive.
NOT not, NOT, and !. The keyword is case insensitive.
Examples:
e_search('abc OR xyz')    # The logical operator is case insensitive.
e_search('abc and (xyz or zzz)')
e_search('abc and not (xyz and not zzz)')
e_search('abc && xyz')    # and
e_search('abc || xyz')    # or
e_search('abc || ! xyz')   # or not
Local operators can also be used between substrings to match occurrences in a string.
e_search('field: (abc OR xyz)')      # The field value contains abc or xyz.
e_search('field: (abc OR not xyz)')  # The field value contains abc or does not contain xyz.
e_search('field: (abc && ! xyz)')     # The field value contains abc and does not contain xyz.

Field check

You can use query strings to check fields.
  • e_search('field: *'): checks whether a field exists.
  • e_search('not field:*'): checks whether a field does not exist.
  • e_search('not field:""'): checks whether a field does not exist.
  • e_search('field: "?"'): checks whether a field exists and its value is not empty.
  • e_search('field==""'): checks whether a field exists and its value is empty.
  • e_search('field~=". +"'): checks whether a field exists and its value is not empty.
  • e_search('not field~=". +"'): checks whether a field does not exist or its value is empty.
  • e_search('not field==""'): checks whether a field does not exist or its value is not empty.