This topic describes all functions that you can use to transform data in Log Service.

Global processing functions

Type Function Description
Flow control functions e_if Performs an operation if a condition is met. Multiple condition-operation pairs can be specified.
  • If a condition is met, the operation that corresponds to it is performed. If the condition is not met, the operation is not performed and the next condition is evaluated.
  • If one operation is performed to delete an event, no more operations can be performed on the event.
e_if_else Performs an operation based on the evaluated result of a condition.
e_switch Performs an operation if a condition is met. Multiple condition-operation pairs can be specified.
  • If a condition is met, the operation that corresponds to it is performed and a result is returned. If the condition is not met, the operation is not performed and the next condition is evaluated.
  • If no specified conditions are met and the default parameter is specified, the default operations that are specified in the default parameter are performed and a result is returned.
  • If one operation is performed to delete an event, no more operations can be performed on the event.
e_compose Combines multiple operations.
  • This function is used to combine multiple operations in the e_if, e_switch, or e_if_else function.
  • The specified operations are performed on an event in sequence and a result is returned.
  • If one operation is performed to delete an event, no more operations can be performed on the event.
Event processing functions e_drop Deletes an event if a condition is met.
e_keep Retains an event if a condition is met.
e_split Splits an event into multiple events based on the value of a specified field. You can use JMESPath to extract the value of the field, and then split the event.
e_output Writes an event to a specified destination Logstore. You can specify parameters such as topic, source, and tags. The event is deleted after it is written to the destination Logstore.
e_coutput Writes an event to a specified destination Logstore. You can specify parameters such as topic, source, and tags. The event is retained after it is written to the destination Logstore. The retained event continues to be transformed.
Field processing functions e_drop_fields Deletes the log fields that meet a specified condition.
e_keep_fields Retains the log fields that meet a specified condition.
e_rename Renames the log fields that meet a specified condition.
Value assignment function e_set Sets the value of a field. Multiple field-value pairs can be specified.
Value extraction functions e_regex Extracts the value of a field in an event by using a regular expression and assigns the value to other fields.
e_json Performs operations on JSON objects in a specified field. You can set the parameters to expand JSON data, extract JSON data with JMESPath expressions, and expand extracted JSON data.
e_kv Extracts key-value pairs of multiple source fields by using the quote parameter.
e_kv_delimit Extracts key-value pairs of multiple source fields by using delimiters.
e_csv Extracts the value of a specified source field, parses the value into multiple values based on the user-defined delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is a comma (,).
e_tsv Extracts the value of a specified source field, parses the value into multiple values based on the user-defined delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is \t.
e_psv Extracts the value of a specified source field, parses the value into multiple values based on the user-defined delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is a vertical line (|).
e_syslogrfc Calculates the values of the facility and severity fields and then returns the corresponding level information. The calculation is based on the value of the priority field in an event by using the RFC protocol that is used by syslog.
e_anchor Extracts the value between the specified start and end positions.
Data mapping and enrichment functions e_dict_map Maps the value of a specified field to a field in a dictionary and returns the value of the matched field.
e_table_map Maps the value of a specified field to a row in a table and returns the value of the field in this row.
e_search_map Maps a search string to a key in a dictionary and returns the value of the matched key.
e_search_table_map Maps a search string to a column in a table and returns the value of the field in this column.
Task configuration function res_local_update Updates the advanced parameters of the current data processing task.

Expression functions

Type Function Description
Event check functions v Extracts the value of a field in an event. If multiple field names are passed to the function, the value of the first field that exists is returned.
e_has Checks whether a field exists.
e_not_has Checks whether a field does not exist.
e_search Searches for specified events in a simplified method.
e_match, e_match_all, and e_match_any Checks whether the value of a field in an event meets the specified condition in an expression.
Operator functions op_if Returns an expression based on a specified condition.
op_ifnull Returns the value of the first expression whose value is not None.
op_coalesce Returns the value of the first expression whose value is not None.
op_nullif If Value 1 is equal to Value 2, the value None is returned. Otherwise, Value 1 is returned.
op_and Invokes the AND operation.
op_not Invokes the NOT operation.
op_or Invokes the OR operation.
op_eq Returns the result that is calculated based on the a==b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
op_ge Returns the result that is calculated based on the a>=b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
op_gt Returns the result that is calculated based on the a>b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
op_le Returns the result that is calculated based on the a<=b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
op_lt Returns the result that is calculated based on the a<b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
op_ne Returns the result that is calculated based on the a! =b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
op_len Calculates the number of characters in a text string. You can use this function in expressions that return strings, tuples, lists, or dictionaries.
op_in Checks whether a string, tuple, list, or dictionary contains a specified element.
op_not_in Checks whether a string, tuple, list, or dictionary does not contain a specified element.
op_slice Truncates a specified string, array, or tuple.
op_index Returns the element that corresponds to the index of a specified string, array, or tuple.
op_add Calculates the sum of multiple values. The values can be strings or numbers.
op_max Returns the maximum value among the values of multiple fields or expressions.
op_min Returns the minimum value among the values of multiple fields or expressions.
Conversion functions ct_int Converts the value of a field or an expression to an integer.
ct_float Converts the value of a field or an expression to a floating-point number.
ct_str Converts the value of a field or an expression to a string.
ct_bool Converts the value of a field or an expression to a Boolean value.
ct_chr Converts the ANSI or Unicode value of a field or an expression to a character.
ct_ord Converts the value of a field or an expression to an ANSI or a Unicode value.
ct_hex Converts the value of a field or an expression to a hexadecimal number.
ct_oct Converts the value of a field or an expression to an octal number.
ct_bin Converts the value of a field or an expression to a binary number.
bin2oct Converts a binary byte string to an octal string.
bin2hex Converts a binary string to a hexadecimal string.
Arithmetic functions op_abs Returns the absolute value of a passed value.
op_div_floor Returns the integer part of the quotient after a division of two passed values.
op_div_true Returns the quotient of two passed values.
op_pow Returns a passed value raised to the power of the other passed value.
op_mul Returns the product of two passed values.
op_neg Returns the opposite number of a passed value.
op_mod Returns the remainder of a passed value divided by the other passed value.
op_sub Returns the difference of two passed values.
op_round Returns a passed value rounded.
op_sum Returns the sum of passed values.
mat_ceil Returns a passed value rounded up to the nearest integer.
mat_exp Returns Euler's number raised to the power of a passed value.
mat_fabs Returns the absolute value of a passed value.
mat_floor Returns a passed value rounded down to the nearest integer.
mat_log Returns the logarithm of a passed value with the other passed value as the base.
mat_log10 Returns the base 10 logarithm of a passed value.
mat_sqrt Returns the square root of a passed value.
mat_degrees Converts radians to degrees.
mat_radians Converts degrees to radians.
mat_sin Returns the sine (in radians) of a passed value.
mat_cos Returns the cosine (in radians) of a passed value.
mat_tan Returns the tangent (in radians) of a passed value.
mat_acos Returns the arc cosine (in radians) of a passed value.
mat_asin Returns the arc sine (in radians) of a passed value.
mat_atan Returns the arc tangent (in radians) of a passed value.
mat_atan2 Returns the arc tangent of X- and Y-coordinates.
mat_atanh Returns the inverse hyperbolic tangent of a passed value.
mat_hypot Returns the Euclidean norm of two passed values.
String functions str_format Formats a string.
str_join Concatenates the elements in a sequence with a specified string to generate a new string.
str_zip Splits the values of two expressions concurrently, and then merges them into a single string.
str_encode Encodes a string by using a specified encoding format.
str_decode Decodes a string by using a specified decoding format.
str_hex_escape_encode Escapes special characters to Chinese characters. Hexadecimal characters are supported.
str_sort Sorts the characters in a string.
str_reverse Reverses the characters in a string.
str_replace Replaces an old string with a new string based on an applicable rule.
str_logtash_config_normalize Converts the data format of the Logstash configuration language to the JSON format.
str_translate Replaces the specified characters of a string with the mapping characters.
str_strip Deletes specified characters from a string.
str_lstrip Deletes specified characters from the start of a string.
str_rstrip Deletes specified characters from the end of a string.
str_lower Converts all uppercase characters in a string to lowercase characters.
str_upper Converts all lowercase characters in a string to uppercase characters.
str_title Capitalizes only the first letter of all words.
str_capitalize Capitalizes only the first letter of a string.
str_swapcase Interchanges the uppercase letters and lowercase letters in a string.
str_count Counts the number of occurrences of a character in a string.
str_find Checks whether an original string contains a specified substring.
str_rfind Returns the position of the last occurrence of a specified character in a string.
str_endswith Checks whether a string ends with a specified suffix.
str_startswith Checks whether a string starts with a specified string.
str_split Splits a string with a specified delimiter.
str_splitlines Splits a string with a line break.
str_partition Splits a string with a specified delimiter into three parts from left to right.
str_rpartition Splits a string with a specified delimiter into three parts from right to left.
str_center Pads a string to a specified length with a specified character.
str_ljust Pads a string to a fixed length from the end with a specified character.
str_rjust Pads a string to a specified length from the start with a specified character.
str_zfill Pads a string to a specified length from the start with 0.
str_expandtabs Converts \t in a string to space characters.
str_isalnum Checks whether a string consists of only letters and digits.
str_isalpha Checks whether a string consists of only letters.
str_isascii Checks whether a string is in the ASCII table.
str_isdecimal Checks whether a string contains only decimal characters.
str_isdigit Checks whether a string consists of only digits.
str_isidentifier Checks whether a string is a valid Python identifier. This function can also check whether a variable name is valid.
str_islower Checks whether a string consists of lowercase letters.
str_isnumeric Checks whether a string consists of digits.
str_isprintable Checks whether all characters in a string are printable characters.
str_isspace Checks whether a string consists of only whitespace characters.
str_istitle Checks whether the first letter of all words in a string are in uppercase and whether other letters in the string are in lowercase.
str_isupper Checks whether all letters in a string are uppercase letters.
str_uuid Generates a random UUID.
Date and time functions dt_parse Converts the value of a time expression to a datetime object.
dt_str Converts the value of a time expression to a string.
dt_parsetimestamp Converts the value of a time expression to a UNIX timestamp.
dt_prop Obtains the specified attribute of the value of a time expression.
dt_now Obtains the current datetime.
dt_today Obtains the current date and excludes the time.
dt_utcnow Obtains the current datetime in the current time zone.
dt_fromtimestamp Converts a UNIX timestamp to a datetime object.
dt_utcfromtimestamp Converts a UNIX timestamp to a datetime object in the current time zone.
dt_strptime Converts a time string to a datetime object.
dt_currentstamp Obtains a UNIX timestamp.
dt_totimestamp Converts a datetime object to a UNIX timestamp.
dt_strftime Converts a datetime object to a string in a specified format.
dt_strftimestamp Converts a UNIX timestamp to a string in a specified format.
dt_truncate Truncates the value of a time expression based on a specified time granularity.
dt_add Modifies the value of a time expression based on a specified time granularity.
dt_MO Offsets the specified time to the date of the previous or next Nth Monday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
dt_TU Offsets the specified time to the date of the previous or next Nth Tuesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
dt_WE Offsets the specified time to the date of the previous or next Nth Wednesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
dt_TH Offsets the specified time to the date of the previous or next Nth Thursday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
dt_FR Offsets the specified time to the date of the previous or next Nth Friday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
dt_SA Offsets the specified time to the date of the previous or next Nth Saturday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
dt_SU Offsets the specified time to the date of the previous or next Nth Sunday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
dt_astimezone Converts the value of a time expression to a datetime object in a specified time zone.
dt_diff Obtains the difference between the values of two time expressions based on a specified time granularity.
Regular expression functions regex_select Extracts a specified value based on a regular expression.
regex_findall Obtains all values that match a regular expression.
regex_match Checks whether a value matches a regular expression.
regex_replace Replaces specified characters in a string based on a regular expression.
regex_split Splits a value into multiple values.
Grok function grok Extracts a specified value based on a regular expression.
Structured data functions json_select Extracts or calculates specified values from a JSON expression based on the JMESPath syntax.
json_parse Converts a JSON text to a JSON object.
xml_to_json Converts XML-formatted data to JSON-formatted data.
pb_to_json Converts Protobuf-formatted data to JSON-formatted data.
geo_parse Parses an IP address and returns the information about the city, province, and country of the IP address.
gzip_compress Compresses specified data, encodes the compressed data by using the Base64 algorithm, and then returns the transformed data.
gzip_decompress Decodes data by using the Base64 algorithm, decompresses the decoded data, and then returns the transformed data.
ip_cidrmatch Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.
ip_version Checks whether the version of an IP address is IPv4 or IPv6.
ip_type Returns the type of an IP address.
ip_makenet Converts an IP address to a CIDR block.
ip_to_format Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.
ip_overlaps Checks whether two CIDR blocks overlap.
Encoding and decoding functions url_encoding Encodes URL data.
url_decoding Decodes URL data.
base64_encoding Encodes data by using the Base64 algorithm.
base64_decoding Decodes data by using the Base64 algorithm.
html_encoding Encodes data in the HTML format.
html_decoding Decodes HTML-encoded data.
md5_encoding Encodes data by using the MD5 algorithm.
sha1_encoding Encodes data by using the SHA1 algorithm.
ip2long Converts an IP address to a value of the long type.
long2ip Converts a value of the long type to an IP address.
Parsing functions ua_parse_device Parses the device information in a User-Agent HTTP header.
ua_parse_os Parses the operating system information in a User-Agent HTTP header.
ua_parse_agent Parses the browser information in a User-Agent HTTP header.
ua_parse_all Parses all information in a User-Agent HTTP header.
List functions lst_make Constructs a list.
lst_insert Inserts values to a list at a specified position.
lst_append Adds values to the end of a list.
lst_delete_at Deletes a value from a list at a specified position.
lst_reverse Reverses the order of values in a list.
lst_get Obtains a value from a list or tuple.
Dictionary functions dct_make Constructs a dictionary.
dct_update Updates a dictionary.
dct_delete Deletes key-value pairs from a dictionary.
dct_keys Obtains the keys of a dictionary.
dct_values Obtains the values of a dictionary.
dct_get Obtains the value that corresponds to a specified key in a dictionary.
Table functions tab_parse_csv Constructs a table from a comma-separated values (CSV) file.
tab_to_dict Constructs a dictionary from a table.
Resource functions res_local Obtains the values of advanced parameters of a data transformation task.
res_rds_mysql Obtains data from a specified database table in ApsaraDB RDS for MySQL. The data can be refreshed at regular intervals.
res_log_logstore_pull Pulls data from another Logstore when a Logstore is being transformed. You can pull data and maintain it in a table in a continuous manner.
res_oss_file Obtains the content of an object in a specified bucket from Object Storage Service (OSS). The object can be refreshed at regular intervals.