This topic describes all functions that are provided by Log Service to transform data.

Global processing functions

Category Function Description
Flow control functions e_if Performs an operation if a condition is met. You can specify multiple condition-operation pairs.
  • If a condition is met, the function performs the operation that corresponds to the condition. If the condition is not met, the function does not perform the operation, but evaluates the next condition.
  • If the function deletes an event by performing an operation, the function no longer performs other operations on the event.
e_if_else Performs an operation based on the evaluation result of a condition.
e_switch Performs an operation if a condition is met. You can specify multiple condition-operation pairs.
  • If a condition is met, the function performs the operation that corresponds to the condition and returns a result. If the condition is not met, the function does not perform the operation, but evaluates the next condition.
  • If no specified conditions are met and the default parameter is configured, the function performs the operation that is specified by the default parameter and returns a result.
  • If the function deletes an event by performing an operation, the function no longer performs other operations on the event.
e_compose Combines multiple operations.
  • The function is commonly used in the e_if, e_switch, or e_if_else function.
  • The function performs specified operations on an event in sequence and returns a result.
  • If the function deletes an event by performing an operation, the function no longer performs other operations on the event.
Event processing functions e_drop Deletes an event if a condition is met.
e_keep Retains an event if a condition is met.
e_split Splits an event into multiple events based on the value of a specified field. You can also use JMESPath to extract the value of the field, and then split the event.
e_output Writes an event to a specified destination Logstore. You can configure parameters such as topic, source, and tags. The event is deleted after the event is written to the destination Logstore.
e_coutput Writes an event to a specified destination Logstore. You can configure parameters such as topic, source, and tags. The event is retained after the event is written to the destination Logstore. The retained event continues to be transformed.
e_to_metric Converts log data to time series data that can be stored in a Metricstore.
Field processing functions v Extracts the value of a field from an event. If multiple field names are passed to the function, the value of the first field that exists in the event is returned.
e_set Adds a field or specifies a new value for an existing field.
e_drop_fields Deletes the log fields that meet a specified condition.
e_keep_fields Retains the log fields that meet a specified condition.
e_pack_fields Packs specified log fields and assigns the log fields as a value to a new field.
e_rename Renames the log fields that meet a specified condition.
Value extraction functions e_regex Extracts the value of a field from an event based on a regular expression and assigns the value to other fields.
e_json Performs operations on JSON objects in a specified field in an event. You can configure the parameters to expand JSON data, extract JSON data by using JMESPath, and expand the extracted JSON data.
e_kv Extracts key-value pairs from multiple source fields and encloses the values of the pairs by using the character that is specified by the quote parameter.
e_kv_delimit Extracts key-value pairs from source fields and separates the key-value pairs by using a specified delimiter.
e_csv Extracts multiple fields from a specified source field by using a specified delimiter and predefined field names. The default delimiter is a comma (,).
e_tsv Extracts multiple fields from a specified source field by using a specified delimiter and predefined field names. The default delimiter is \t.
e_psv Extracts multiple fields from a specified source field by using a specified delimiter and predefined field names. The default delimiter is a vertical bar(|).
e_syslogrfc Calculates the values of the facility and severity fields and returns level information, which is indicated by the facilitylabel field. The function calculates the values based on the value of the priority field and the specified syslog protocol.
e_anchor Extracts strings by using the defined anchor_rules.
Mapping and enrichment functions e_dict_map Maps the value of a source field to a value in a specified dictionary and returns a new field.
e_table_map Maps the value of a source field to a row in a specified table and returns a new field.
e_search_dict_map Maps data based on a specified dictionary and returns new fields. The dictionary consists of keys that are query strings and values for the keys.
e_search_table_map Maps data based on a specified table and returns new fields. The table consists of a column whose values are query strings and the rows for the column.
Value-added content functions e_threat_intelligence Obtains threat intelligence from a log field and assigns the threat intelligence as a value to a new field.

Expression functions

Category Function Description
Event check functions e_has Checks whether a field exists.
e_not_has Checks whether a field does not exist.
e_search Searches for events in a simplified manner.
e_match, e_match_all, and e_match_any Checks whether the value of a field in an event matches a specified expression.
Operator functions op_if Returns an expression based on a specified condition.
op_ifnull Returns the value of the first expression whose value is not None.
op_coalesce Returns the value of the first expression whose value is not None.
op_nullif Returns the value None if the value of Expression 1 is equal to the value of Expression 2. If the values of Expression 1 and Expression 2 are different, the value of Expression 1 is returned.
op_and Invokes the AND operation.
op_not Invokes the NOT operation.
op_or Invokes the OR operation.
op_eq Returns the result that is calculated based on the a==b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
op_ge Returns the result that is calculated based on the a>=b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
op_gt Returns the result that is calculated based on the a>b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
op_le Returns the result that is calculated based on the a<=b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
op_lt Returns the result that is calculated based on the a<b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
op_ne Returns the result that is calculated based on the a!=b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
op_len Calculates the number of characters in a text string. You can use this function in strings and expressions that return tuples, lists, or dictionaries.
op_in Checks whether a string, tuple, list, or dictionary contains a specified element.
op_not_in Checks whether a string, tuple, list, or dictionary does not contain a specified element.
op_slice Extracts strings from a specified string, array, or tuple.
op_index Returns the element that corresponds to the index of a specified string, array, or tuple.
op_add Calculates the sum of multiple values. The values can be strings or numbers.
op_max Returns the largest value among the values of multiple fields or expressions.
op_min Returns the smallest value among the values of multiple fields or expressions.
Conversion functions ct_int Converts the value of a field or an expression to an integer.
ct_float Converts the value of a field or an expression to a floating-point number.
ct_str Converts the value of a field or an expression to a string.
ct_bool Converts the value of a field or an expression to a Boolean value.
ct_chr Converts the ANSI or Unicode value of a field or an expression to a character.
ct_ord Converts the value of a field or an expression to an ANSI or Unicode value.
ct_hex Converts the value of a field or an expression to a hexadecimal number.
ct_oct Converts the value of a field or an expression to an octal number.
ct_bin Converts the value of a field or an expression to a binary number.
bin2oct Converts a binary byte string to an octal string.
bin2hex Converts a binary byte string to a hexadecimal string.
Arithmetic functions op_abs Returns the absolute value of an input value.
op_div_floor Returns the integer part of the quotient of two input values.
op_div_true Returns the quotient of two input values.
op_pow Raises an input value to a specified power.
op_mul Returns the product of two input values.
op_neg Returns the opposite number of an input value.
op_mod Returns the remainder of an input value divided by the other input value.
op_sub Returns the difference of two input values.
op_round Rounds an input value.
op_sum Returns the sum of input values.
mat_ceil Rounds an input value up to the nearest integer.
mat_exp Raises e to a specified power.
mat_fabs Returns the absolute value of an input value.
mat_floor Rounds an input value down to the nearest integer.
mat_log Returns the logarithm of an input value with the base specified by the other input value.
mat_log10 Returns the base-10 logarithm of an input value.
mat_sqrt Returns the square root of an input value.
mat_degrees Converts radians to degrees.
mat_radians Converts degrees to radians.
mat_sin Returns the sine of an input value in radians.
mat_cos Returns the cosine of an input value in radians.
mat_tan Returns the tangent of an input value in radians.
mat_acos Returns the arc cosine of an input value in radians.
mat_asin Returns the arc sine of an input value in radians.
mat_atan Returns the arc tangent of an input value in radians.
mat_atan2 Returns the arc tangent of X and Y coordinates.
mat_atanh Returns the inverse hyperbolic tangent of an input value.
mat_hypot Returns the Euclidean norm of two input values.
String functions str_format Formats strings.
str_join Concatenates the elements in a sequence with a specified connector to generate a new string.
str_zip Concurrently splits two values or strings that are returned by expressions and combines the results to a string.
str_encode Encodes a string.
str_decode Decodes a string.
str_hex_escape_encode Escapes special characters. The function can escape hexadecimal characters to Chinese characters.
str_sort Sorts the characters in a string.
str_reverse Reverses the characters in a string.
str_replace Replaces an old string with a new string based on a specified rule.
str_logtash_config_normalize Converts the data in the Logstash configuration language to the JSON format.
str_translate Replaces specified characters in a string with mapping characters.
str_strip Deletes specified characters from a string.
str_lstrip Deletes specified characters from the start of a string.
str_rstrip Deletes specified characters from the end of a string.
str_lower Converts all uppercase letters in a string to lowercase letters.
str_upper Converts all lowercase letters in a string to uppercase letters.
str_title Capitalizes the first letter of each word in a string and converts the other letters in the string to lowercase letters.
str_capitalize Capitalizes the first letter of a string and converts the other letters in the string to lowercase letters.
str_swapcase Interchanges the uppercase letters and lowercase letters in a string.
str_count Counts the number of occurrences of a character in a string.
str_find Checks whether a string contains a specified substring.
str_rfind Returns the position of the last occurrence of a specified character in a string.
str_endswith Checks whether a string ends with a specified suffix.
str_startswith Checks whether a string starts with a specified substring.
str_split Splits a string by using a specified delimiter.
str_splitlines Splits a string by using a line feed.
str_partition Splits a string into three parts from left to right by using a specified delimiter.
str_rpartition Splits a string into three parts from right to left by using a specified delimiter.
str_center Pads a string to a specified length by using a specified character.
str_ljust Right pads a string to a specified length by using a specified character.
str_rjust Left pads a string to a specified length by using a specified character.
str_zfill Left pads a string to a specified length by using 0.
str_expandtabs Converts \t in a string to spaces.
str_isalnum Checks whether a string contains only letters and digits.
str_isalpha Checks whether a string contains only letters.
str_isascii Checks whether a string is in the ASCII table.
str_isdecimal Checks whether a string contains only decimal characters.
str_isdigit Checks whether a string contains only digits.
str_isidentifier Checks whether a string is a valid Python identifier or checks whether a variable name is valid.
str_islower Checks whether a string contains lowercase letters.
str_isnumeric Checks whether a string contains digits.
str_isprintable Checks whether all characters in a string are printable characters.
str_isspace Checks whether a string contains only spaces.
str_istitle Checks whether the first letter of each word in a string is in uppercase and whether the other letters in the string are in lowercase.
str_isupper Checks whether all letters in a string are in uppercase.
str_uuid Generates a random UUID.
Date and time functions dt_parse Converts a value or the value of a time expression to a datetime object.
dt_str Converts a value or the value of a time expression to a string.
dt_parsetimestamp Converts a value or the value of a time expression to a UNIX timestamp.
dt_prop Returns the specified attribute of a value or the value of a time expression.
dt_now Returns the current date and time.
dt_today Returns only the current date.
dt_utcnow Returns the current date and time in the current time zone.
dt_fromtimestamp Converts a UNIX timestamp to a datetime object.
dt_utcfromtimestamp Converts a UNIX timestamp to a datetime object in the current time zone.
dt_strptime Parses a time string into a datetime object.
dt_currentstamp Returns a UNIX timestamp.
dt_totimestamp Converts a datetime object to a UNIX timestamp.
dt_strftime Converts a datetime object to a string in a specified format.
dt_strftimestamp Converts a UNIX timestamp to a string in a specified format.
dt_truncate Extracts a time value from a value or the value of a time expression based on a specified time granularity.
dt_add Changes a value or the value of a time expression based on a specified time granularity.
dt_MO Offsets a specified time to the date of the previous or following Nth Monday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. If you want to pass a negative integer, use op_neg(Positive integer).
dt_TU Offsets a specified time to the date of the previous or following Nth Tuesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. If you want to pass a negative integer, use op_neg(Positive integer).
dt_WE Offsets a specified time to the date of the previous or following Nth Wednesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. If you want to pass a negative integer, use op_neg(Positive integer).
dt_TH Offsets a specified time to the date of the previous or following Nth Thursday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. If you want to pass a negative integer, use op_neg(Positive integer).
dt_FR Offsets a specified time to the date of the previous or following Nth Friday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. If you want to pass a negative integer, use op_neg(Positive integer).
dt_SA Offsets a specified time to the date of the previous or following Nth Saturday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. If you want to pass a negative integer, use op_neg(Positive integer).
dt_SU Offsets a specified time to the date of the previous or following Nth Sunday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. If you want to pass a negative integer, use op_neg(Positive integer).
dt_astimezone Converts a value or the value of a time expression to a datetime object in a specified time zone.
dt_diff Returns the difference between two values or the values of two time expressions based on a specified time granularity.
Regular expression functions regex_select Extracts a specified value based on a regular expression.
regex_findall Extracts all values that match a regular expression.
regex_match Checks whether a value matches a regular expression.
regex_replace Replaces specified characters in a string based on a regular expression.
regex_split Splits a string into an array of strings.
Grok function grok Extracts a specified value based on a regular expression.
Structured data functions json_select Extracts or calculates specified values from a JSON expression by using JMESPath.
json_parse Parses a specified value into a JSON object.
xml_to_json Converts XML data to JSON data and expands the data.
gzip_compress Compresses data, encodes the compressed data by using the Base64 algorithm, and then returns the result.
gzip_decompress Decodes data by using the Base64 algorithm, decompresses the decoded data, and then returns the result.
IP address parsing functions geo_parse Parses an IP address into the information about the city, province, and country of the IP address.
ip_cidrmatch Checks whether an IP address belongs to a CIDR block.
ip_version Checks whether the version of an IP address is IPv4 or IPv6.
ip_type Identifies the type of an IP address and checks whether the type of the IP address is private or public.
ip_makenet Converts an IP address to a CIDR block.
ip_to_format Converts a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.
ip_overlaps Checks whether two CIDR blocks overlap.
Encoding and decoding functions url_encoding Encodes URL data.
url_decoding Decodes URL data.
base64_encoding Encodes data by using the Base64 algorithm.
base64_decoding Decodes data by using the Base64 algorithm.
html_encoding Encodes data in the HTML format.
html_decoding Decodes data that is encoded in the HTML format.
md5_encoding Encodes data by using the MD5 algorithm.
sha1_encoding Encodes data by using the SHA1 algorithm.
ip2long Converts an IP address to a value of the long type.
long2ip Converts a value of the long type to an IP address.
aes_encrypt Encrypts data by using the AES algorithm.
aes_decrypt Decrypts data by using the AES algorithm.
Parsing functions ua_parse_device Parses User-Agent and returns the device information.
ua_parse_os Parses User-Agent and returns the operating system information.
ua_parse_agent Parses User-Agent and returns the browser information.
ua_parse_all Parses User-Agent and returns all information.
List functions lst_make Creates a list.
lst_insert Inserts elements to a specified position in a list.
lst_append Adds elements to the end of a list.
lst_delete_at Deletes the element at a specified position from a list.
lst_reverse Reverses the values in a list.
lst_get Returns an element of a list or a tuple.
Dictionary functions dct_make Constructs a dictionary.
dct_update Updates a dictionary.
dct_delete Deletes key-value pairs from a dictionary.
dct_keys Returns the keys of a dictionary.
dct_values Returns the values of a dictionary.
dct_get Returns the value that corresponds to a specified key in a dictionary.
Table functions tab_parse_csv Constructs a table from CSV text.
tab_to_dict Constructs a dictionary from a table.
Resource functions res_local Returns the value of an advanced parameter that is specified in a data transformation task.
res_rds_mysql Returns the data of a specified database table in ApsaraDB RDS for MySQL. The data can be refreshed at regular intervals.
res_log_logstore_pull Pulls data from another Logstore when data in a Logstore is being transformed. You can pull data and maintain table data in a continuous manner.
res_oss_file Returns the content of an object in a specified bucket of Object Storage Service (OSS). The content can be refreshed at regular intervals.