Function overview

This topic describes all functions that you can use to transform data in Log Service.

Global processing functions

Type	Function	Description
Flow control functions	e_if	Performs an operation if a condition is met. Multiple condition-operation pairs can be specified. If a condition is met, the operation that corresponds to it is performed. If the condition is not met, the operation is not performed and the next condition is evaluated. If one operation is performed to delete an event, no more operations can be performed on the event.
	e_if_else	Performs an operation based on the evaluated result of a condition.
	e_switch	Performs an operation if a condition is met. Multiple condition-operation pairs can be specified. If a condition is met, the operation that corresponds to it is performed and a result is returned. If the condition is not met, the operation is not performed and the next condition is evaluated. If no specified conditions are met and the default parameter is specified, the default operations that are specified in the default parameter are performed and a result is returned. If one operation is performed to delete an event, no more operations can be performed on the event.
	e_compose	Combines multiple operations. This function is used to combine multiple operations in the e_if, e_switch, or e_if_else function. The specified operations are performed on an event in sequence and a result is returned. If one operation is performed to delete an event, no more operations can be performed on the event.
Event processing functions	e_drop	Deletes an event if a condition is met.
	e_keep	Retains an event if a condition is met.
	e_split	Splits an event into multiple events based on the value of a specified field. You can use JMESPath to extract the value of the field, and then split the event.
	e_output	Writes an event to a specified destination Logstore. You can specify parameters such as topic, source, and tags. The event is deleted after it is written to the destination Logstore.
	e_coutput	Writes an event to a specified destination Logstore. You can specify parameters such as topic, source, and tags. The event is retained after it is written to the destination Logstore. The retained event continues to be transformed.
Field processing functions	e_drop_fields	Deletes the log fields that meet a specified condition.
	e_keep_fields	Retains the log fields that meet a specified condition.
	e_rename	Renames the log fields that meet a specified condition.
Value assignment function	e_set	Sets the value of a field. Multiple field-value pairs can be specified.
Value extraction functions	e_regex	Extracts the value of a field in an event by using a regular expression and assigns the value to other fields.
	e_json	Performs operations on JSON objects in a specified field. You can set the parameters to expand JSON data, extract JSON data with JMESPath expressions, and expand extracted JSON data.
	e_kv	Extracts key-value pairs of multiple source fields by using the quote parameter.
	e_kv_delimit	Extracts key-value pairs of multiple source fields by using delimiters.
	e_csv	Extracts the value of a specified source field, parses the value into multiple values based on the user-defined delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is a comma (,).
	e_tsv	Extracts the value of a specified source field, parses the value into multiple values based on the user-defined delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is \t.
	e_psv	Extracts the value of a specified source field, parses the value into multiple values based on the user-defined delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is a vertical line (|).
	e_syslogrfc	Calculates the values of the facility and severity fields and then returns the corresponding level information. The calculation is based on the value of the priority field in an event by using the RFC protocol that is used by syslog.
	e_anchor	Extracts the value between the specified start and end positions.
Data mapping and enrichment functions	e_dict_map	Maps the value of a specified field to a field in a dictionary and returns the value of the matched field.
	e_table_map	Maps the value of a specified field to a row in a table and returns the value of the field in this row.
	e_search_map	Maps a search string to a key in a dictionary and returns the value of the matched key.
	e_search_table_map	Maps a search string to a column in a table and returns the value of the field in this column.
Task configuration function	res_local_update	Updates the advanced parameters of the current data processing task.

Expression functions

Type	Function	Description
Event check functions	v	Extracts the value of a field in an event. If multiple field names are passed to the function, the value of the first field that exists is returned.
	e_has	Checks whether a field exists.
	e_not_has	Checks whether a field does not exist.
	e_search	Searches for specified events in a simplified method.
	e_match, e_match_all, and e_match_any	Checks whether the value of a field in an event meets the specified condition in an expression.
Operator functions	op_if	Returns an expression based on a specified condition.
	op_ifnull	Returns the value of the first expression whose value is not None.
	op_coalesce	Returns the value of the first expression whose value is not None.
	op_nullif	If Value 1 is equal to Value 2, the value None is returned. Otherwise, Value 1 is returned.
	op_and	Invokes the AND operation.
	op_not	Invokes the NOT operation.
	op_or	Invokes the OR operation.
	op_eq	Returns the result that is calculated based on the a==b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
	op_ge	Returns the result that is calculated based on the a>=b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
	op_gt	Returns the result that is calculated based on the a>b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
	op_le	Returns the result that is calculated based on the a<=b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
	op_lt	Returns the result that is calculated based on the a<b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
	op_ne	Returns the result that is calculated based on the a! =b condition. The values of the a and b parameters must be of the same data type, such as string, number, or list.
	op_len	Calculates the number of characters in a text string. You can use this function in expressions that return strings, tuples, lists, or dictionaries.
	op_in	Checks whether a string, tuple, list, or dictionary contains a specified element.
	op_not_in	Checks whether a string, tuple, list, or dictionary does not contain a specified element.
	op_slice	Truncates a specified string, array, or tuple.
	op_index	Returns the element that corresponds to the index of a specified string, array, or tuple.
	op_add	Calculates the sum of multiple values. The values can be strings or numbers.
	op_max	Returns the maximum value among the values of multiple fields or expressions.
	op_min	Returns the minimum value among the values of multiple fields or expressions.
Conversion functions	ct_int	Converts the value of a field or an expression to an integer.
	ct_float	Converts the value of a field or an expression to a floating-point number.
	ct_str	Converts the value of a field or an expression to a string.
	ct_bool	Converts the value of a field or an expression to a Boolean value.
	ct_chr	Converts the ANSI or Unicode value of a field or an expression to a character.
	ct_ord	Converts the value of a field or an expression to an ANSI or a Unicode value.
	ct_hex	Converts the value of a field or an expression to a hexadecimal number.
	ct_oct	Converts the value of a field or an expression to an octal number.
	ct_bin	Converts the value of a field or an expression to a binary number.
	bin2oct	Converts a binary byte string to an octal string.
	bin2hex	Converts a binary string to a hexadecimal string.
Arithmetic functions	op_abs	Returns the absolute value of a passed value.
	op_div_floor	Returns the integer part of the quotient after a division of two passed values.
	op_div_true	Returns the quotient of two passed values.
	op_pow	Returns a passed value raised to the power of the other passed value.
	op_mul	Returns the product of two passed values.
	op_neg	Returns the opposite number of a passed value.
	op_mod	Returns the remainder of a passed value divided by the other passed value.
	op_sub	Returns the difference of two passed values.
	op_round	Returns a passed value rounded.
	op_sum	Returns the sum of passed values.
	mat_ceil	Returns a passed value rounded up to the nearest integer.
	mat_exp	Returns Euler's number raised to the power of a passed value.
	mat_fabs	Returns the absolute value of a passed value.
	mat_floor	Returns a passed value rounded down to the nearest integer.
	mat_log	Returns the logarithm of a passed value with the other passed value as the base.
	mat_log10	Returns the base 10 logarithm of a passed value.
	mat_sqrt	Returns the square root of a passed value.
	mat_degrees	Converts radians to degrees.
	mat_radians	Converts degrees to radians.
	mat_sin	Returns the sine (in radians) of a passed value.
	mat_cos	Returns the cosine (in radians) of a passed value.
	mat_tan	Returns the tangent (in radians) of a passed value.
	mat_acos	Returns the arc cosine (in radians) of a passed value.
	mat_asin	Returns the arc sine (in radians) of a passed value.
	mat_atan	Returns the arc tangent (in radians) of a passed value.
	mat_atan2	Returns the arc tangent of X- and Y-coordinates.
	mat_atanh	Returns the inverse hyperbolic tangent of a passed value.
	mat_hypot	Returns the Euclidean norm of two passed values.
String functions	str_format	Formats a string.
	str_join	Concatenates the elements in a sequence with a specified string to generate a new string.
	str_zip	Splits the values of two expressions concurrently, and then merges them into a single string.
	str_encode	Encodes a string by using a specified encoding format.
	str_decode	Decodes a string by using a specified decoding format.
	str_hex_escape_encode	Escapes special characters to Chinese characters. Hexadecimal characters are supported.
	str_sort	Sorts the characters in a string.
	str_reverse	Reverses the characters in a string.
	str_replace	Replaces an old string with a new string based on an applicable rule.
	str_logtash_config_normalize	Converts the data format of the Logstash configuration language to the JSON format.
	str_translate	Replaces the specified characters of a string with the mapping characters.
	str_strip	Deletes specified characters from a string.
	str_lstrip	Deletes specified characters from the start of a string.
	str_rstrip	Deletes specified characters from the end of a string.
	str_lower	Converts all uppercase characters in a string to lowercase characters.
	str_upper	Converts all lowercase characters in a string to uppercase characters.
	str_title	Capitalizes only the first letter of all words.
	str_capitalize	Capitalizes only the first letter of a string.
	str_swapcase	Interchanges the uppercase letters and lowercase letters in a string.
	str_count	Counts the number of occurrences of a character in a string.
	str_find	Checks whether an original string contains a specified substring.
	str_rfind	Returns the position of the last occurrence of a specified character in a string.
	str_endswith	Checks whether a string ends with a specified suffix.
	str_startswith	Checks whether a string starts with a specified string.
	str_split	Splits a string with a specified delimiter.
	str_splitlines	Splits a string with a line break.
	str_partition	Splits a string with a specified delimiter into three parts from left to right.
	str_rpartition	Splits a string with a specified delimiter into three parts from right to left.
	str_center	Pads a string to a specified length with a specified character.
	str_ljust	Pads a string to a fixed length from the end with a specified character.
	str_rjust	Pads a string to a specified length from the start with a specified character.
	str_zfill	Pads a string to a specified length from the start with 0.
	str_expandtabs	Converts \t in a string to space characters.
	str_isalnum	Checks whether a string consists of only letters and digits.
	str_isalpha	Checks whether a string consists of only letters.
	str_isascii	Checks whether a string is in the ASCII table.
	str_isdecimal	Checks whether a string contains only decimal characters.
	str_isdigit	Checks whether a string consists of only digits.
	str_isidentifier	Checks whether a string is a valid Python identifier. This function can also check whether a variable name is valid.
	str_islower	Checks whether a string consists of lowercase letters.
	str_isnumeric	Checks whether a string consists of digits.
	str_isprintable	Checks whether all characters in a string are printable characters.
	str_isspace	Checks whether a string consists of only whitespace characters.
	str_istitle	Checks whether the first letter of all words in a string are in uppercase and whether other letters in the string are in lowercase.
	str_isupper	Checks whether all letters in a string are uppercase letters.
	str_uuid	Generates a random UUID.
Date and time functions	dt_parse	Converts the value of a time expression to a datetime object.
	dt_str	Converts the value of a time expression to a string.
	dt_parsetimestamp	Converts the value of a time expression to a UNIX timestamp.
	dt_prop	Obtains the specified attribute of the value of a time expression.
	dt_now	Obtains the current datetime.
	dt_today	Obtains the current date and excludes the time.
	dt_utcnow	Obtains the current datetime in the current time zone.
	dt_fromtimestamp	Converts a UNIX timestamp to a datetime object.
	dt_utcfromtimestamp	Converts a UNIX timestamp to a datetime object in the current time zone.
	dt_strptime	Converts a time string to a datetime object.
	dt_currentstamp	Obtains a UNIX timestamp.
	dt_totimestamp	Converts a datetime object to a UNIX timestamp.
	dt_strftime	Converts a datetime object to a string in a specified format.
	dt_strftimestamp	Converts a UNIX timestamp to a string in a specified format.
	dt_truncate	Truncates the value of a time expression based on a specified time granularity.
	dt_add	Modifies the value of a time expression based on a specified time granularity.
	dt_MO	Offsets the specified time to the date of the previous or next Nth Monday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_TU	Offsets the specified time to the date of the previous or next Nth Tuesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_WE	Offsets the specified time to the date of the previous or next Nth Wednesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_TH	Offsets the specified time to the date of the previous or next Nth Thursday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_FR	Offsets the specified time to the date of the previous or next Nth Friday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_SA	Offsets the specified time to the date of the previous or next Nth Saturday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_SU	Offsets the specified time to the date of the previous or next Nth Sunday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_astimezone	Converts the value of a time expression to a datetime object in a specified time zone.
	dt_diff	Obtains the difference between the values of two time expressions based on a specified time granularity.
Regular expression functions	regex_select	Extracts a specified value based on a regular expression.
	regex_findall	Obtains all values that match a regular expression.
	regex_match	Checks whether a value matches a regular expression.
	regex_replace	Replaces specified characters in a string based on a regular expression.
	regex_split	Splits a value into multiple values.
Grok function	grok	Extracts a specified value based on a regular expression.
Structured data functions	json_select	Extracts or calculates specified values from a JSON expression based on the JMESPath syntax.
	json_parse	Converts a JSON text to a JSON object.
	xml_to_json	Converts XML-formatted data to JSON-formatted data.
	pb_to_json	Converts Protobuf-formatted data to JSON-formatted data.
	geo_parse	Parses an IP address and returns the information about the city, province, and country of the IP address.
	gzip_compress	Compresses specified data, encodes the compressed data by using the Base64 algorithm, and then returns the transformed data.
	gzip_decompress	Decodes data by using the Base64 algorithm, decompresses the decoded data, and then returns the transformed data.
	ip_cidrmatch	Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.
	ip_version	Checks whether the version of an IP address is IPv4 or IPv6.
	ip_type	Returns the type of an IP address.
	ip_makenet	Converts an IP address to a CIDR block.
	ip_to_format	Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.
	ip_overlaps	Checks whether two CIDR blocks overlap.
Encoding and decoding functions	url_encoding	Encodes URL data.
	url_decoding	Decodes URL data.
	base64_encoding	Encodes data by using the Base64 algorithm.
	base64_decoding	Decodes data by using the Base64 algorithm.
	html_encoding	Encodes data in the HTML format.
	html_decoding	Decodes HTML-encoded data.
	md5_encoding	Encodes data by using the MD5 algorithm.
	sha1_encoding	Encodes data by using the SHA1 algorithm.
	ip2long	Converts an IP address to a value of the long type.
	long2ip	Converts a value of the long type to an IP address.
Parsing functions	ua_parse_device	Parses the device information in a User-Agent HTTP header.
	ua_parse_os	Parses the operating system information in a User-Agent HTTP header.
	ua_parse_agent	Parses the browser information in a User-Agent HTTP header.
	ua_parse_all	Parses all information in a User-Agent HTTP header.
List functions	lst_make	Constructs a list.
	lst_insert	Inserts values to a list at a specified position.
	lst_append	Adds values to the end of a list.
	lst_delete_at	Deletes a value from a list at a specified position.
	lst_reverse	Reverses the order of values in a list.
	lst_get	Obtains a value from a list or tuple.
Dictionary functions	dct_make	Constructs a dictionary.
	dct_update	Updates a dictionary.
	dct_delete	Deletes key-value pairs from a dictionary.
	dct_keys	Obtains the keys of a dictionary.
	dct_values	Obtains the values of a dictionary.
	dct_get	Obtains the value that corresponds to a specified key in a dictionary.
Table functions	tab_parse_csv	Constructs a table from a comma-separated values (CSV) file.
	tab_to_dict	Constructs a dictionary from a table.
Resource functions	res_local	Obtains the values of advanced parameters of a data transformation task.
	res_rds_mysql	Obtains data from a specified database table in ApsaraDB RDS for MySQL. The data can be refreshed at regular intervals.
	res_log_logstore_pull	Pulls data from another Logstore when a Logstore is being transformed. You can pull data and maintain it in a table in a continuous manner.
	res_oss_file	Obtains the content of an object in a specified bucket from Object Storage Service (OSS). The object can be refreshed at regular intervals.