Query strings are used in the domain-specific language (DSL) for Log Service to filter log data in an efficient manner and simplify condition matching. This topic describes the rules for specifying query strings.

Functions

The following table describes the functions that use query strings.
Category Function Scenario
Event check functions e_search Query strings are used to check whether the value of a field in an event meets specified conditions.
Resource functions res_log_logstore_pull Query strings are used to configure a field blacklist or a field whitelist to filter data from a Logstore and return a table.
res_rds_mysql Query strings are used to configure a field blacklist or a field whitelist to filter data from a specified table of an ApsaraDB RDS for MySQL database and return a table.
Event mapping functions e_search_table_map and e_search_dict_map Query strings are used to match key-value pairs in a dictionary.

Features

The following table lists the search features that support field search and full-text search.
Feature Field search Full-text search
Search for substrings Supported Supported
Search for strings by using wildcard characters, which support asterisks (*) and question marks (?) Supported Supported
Exact match Supported Not supported
Search for strings by using regular expressions Supported Not supported
Search for strings by comparing numeric ranges Supported Not supported
Search for strings by comparing numeric values Supported Not supported
Search for strings by using logical operators (AND, OR, and NOT), or a combination of these operators Supported Supported

Escape special characters

Special characters, such as asterisks (*) and backslashes (\), must be escaped in query strings.

  • Escape special characters in a field name
    Field names cannot be enclosed in double quotation marks (""). Special characters in a field name must be escaped by using backslashes (\). Examples:
    • \*\(1+1\)\?: abc. Special characters are escaped by using backslashes (\).
    • __tag__\:__container_name__: abc. Special characters are escaped by using backslashes (\).
    • "content": abc. In this example, the field name is invalid. The field name cannot be enclosed in double quotation marks ("").
  • Escape special characters in a field value
    • To query a field value that contains special characters such as quotation marks (") and backslashes (\), you must escape the special characters by using backslashes (\). Example: content: "abc\"xy\\z".
      Note A field value must be enclosed in double quotation marks (""). You can use single quotation marks('') to enclose the string and double quotation marks ("") to enclose the field value. For example, e_search("domain: '/url/test.jsp'") is invalid, and e_search('domain: "/url/test.jsp"') is valid.
    • To query a field value that contains special characters such as asterisks (*) and question marks (?), you must escape the special characters by using backslashes (\). If you do not escape the special characters by using backslashes (\), the special characters are used as wildcard characters for matching.
    • To query a field value that contains only letters, digits, underscores (_), hyphens (-), asterisks (*), and question marks (?), you do not need to enclose the field value in double quotation marks (""). When other characters are used, you must enclose the field value in double quotation marks (""). Examples:
      • status: "\*\?()[]:=". The field value is enclosed in double quotation marks (""). The asterisk (*) and question mark (?) are escaped by using backslashes (\). Characters other than the asterisk (*) and question mark (?) are not escaped in the field value.
      • content: ()[]:=. The field value is invalid. The field value must be enclosed in double quotation marks ("").
      • status: active\*test and status: active\?test. The field values contain only letters, an asterisk (*) and a question mark (?). The field values do not need to be enclosed in double quotation marks (""). The asterisk (*) and question mark (?) in the field values are escaped by using backslashes (\).

Search for substrings

  • Full-text search

    Search for substrings in all fields.

    • Syntax
      e_search('substring')
    • Examples
      • e_search('"error"'): searches for a substring.
      • e_search('"active error"'): searches for a substring that contains a space.
      • e_search('active error'): searches for multiple substrings. The logical operator OR is used by default.
  • Field search

    Search for substrings in specific fields.

    • Syntax
      e_search('...')
    • Examples
      • e_search('status: active'): searches for a substring.
      • e_search('author: "john smith"'): searches for a substring that contains a space.
      Note e_search('field: active error'): searches for active in the field field or searches for error in all fields. In this example, the query string is equivalent to field:active OR "error".

Search for strings by using wildcard characters

An asterisk (*) specifies zero or multiple characters. A question mark (?) specifies one character or one wide character.

  • Full-text search
    Search for substrings in all fields.
    • Syntax
      e_search('substring')
    • Examples
      • e_search('active*test'). The asterisk (*) is used to match zero or multiple characters. The query string does not need to be enclosed in double quotation marks ("") because the query string contains only letters and an asterisk (*).
      • e_search('*error occurs'). The asterisk (*) is used to match zero or multiple characters. For example, the error occurs and critical error occurs strings can be matched.
      • e_search('active?good'). The question mark (?) is used to match one character. The query string does not need to be enclosed in double quotation marks ("") because the query string contains only letters and a question mark (?).
      • e_search('ac*tive?good'). The query string is used to perform an exact match by using an asterisk (*) and a question mark (?).
      • e_search('ac*tive??go*od'). The query string is used to perform an exact match by using multiple asterisks (*) and question marks (?).
  • Field search
    Search for substrings in specific fields.
    • Syntax
      e_search('field name: substring')
    • Examples
      • e_search('status: active*test'). The asterisk (*) is used to match zero or multiple characters.
      • e_search('status: active?good'): The question mark (?) is used to match one character.

Exact match

In exact match, the entire field value is matched.
  • Syntax
    e_search('field name==string that must be exactly matched')
  • Examples
    • e_search('author== "john smith"'). The value of the author field must be john smith.
    • e_search('status== ac*tive?good'). The query string contains wildcard characters and is used for exact match.

Search for strings by using regular expressions

Regular expressions are more efficient than wildcard characters in matching.
  • Syntax
    e_search('field name~=regular expression')
    Note
    • Regular expressions may contain backslashes (\). We recommend that you use r to prevent the system from escaping the backslashes (\).
    • By default, Log Service performs fuzzy match. To enable exact match, you must specify a regular expression that includes a caret (^) as a prefix and a dollar sign ($) as a suffix.
  • Examples
    • e_search('status~= "\d+"'). The value of the status field contains digits.
    • e_search('status~= "^\d+$"'). The value of the status field is a number.

Search for strings by comparing numeric values or numeric ranges

You can search for field values by comparing field values with specified numeric values or numeric ranges.
  • Numeric value comparison

    You can compare field values with specified numeric values by using the following operators. The operators are greater-than (>), greater-than-or-equal-to (>=), equal-to (=), less-than (<), and less-than-or-equal-to (<=).

    e_search('age >= 18')  #  >=18
    e_search('age > 18')   #  > 18
    e_search('age = 18')   #  = 18
    e_search('age <= 18')  #  <=18
    e_search('age < 18')   #  < 18
  • Numeric range comparison

    You can search for field values that are within a closed interval. An asterisk (*) can be used to specify an infinite interval.

    e_search('count: [100, 200]') # >=100 and  <=200
    e_search('count: [*, 200]')   # <=200
    e_search('count: [200, *]')   # >=200

Search for strings by judging logical relationships

Logical operators can be used among multiple search conditions. Parentheses () are used to nest search conditions.
Logical operator Keyword
AND and, AND, and &&. The keywords are not case-sensitive.
OR or and OR. The keywords are not case-sensitive.
NOT not, NOT, and !. The keywords are not case-sensitive.
Examples:
e_search('abc OR xyz')    # The logical operator is not case-sensitive.
e_search('abc and (xyz or zzz)')
e_search('abc and not (xyz and not zzz)')
e_search('abc && xyz')    # and
e_search('abc || xyz')    # or
e_search('abc || !xyz')   # or not
Local operators can also be used to match substrings.
e_search('field: (abc OR xyz)')      # The field value contains abc or xyz.
e_search('field: (abc OR not xyz)')  # The field value contains abc or does not contain xyz.
e_search('field: (abc && !xyz)')     # The field value contains abc and does not contain xyz.

Field check

You can use query strings to check fields.
  • e_search('field: *'): checks whether a field exists.
  • e_search('not field:*'): checks whether a field does not exist.
  • e_search('not field:""'): checks whether a field does not exist.
  • e_search('field: "?"'): checks whether a field exists and whether the field is not empty.
  • e_search('field==""'): checks whether a field exists and whether the field is empty.
  • e_search('field~=".+"'): checks whether a field exists and whether the field is not empty.
  • e_search('not field~=".+"'): checks whether a field does not exist or whether the field is empty.
  • e_search('not field==""'): checks whether a field does not exist or whether the field is not empty.