Function overview

This topic describes all functions that are provided by Log Service to transform data.

Global processing functions

Type	Function	Description
Flow control functions	e_if	Performs an operation if a condition is met. Multiple condition-operation pairs can be specified. If a condition is met, the operation that corresponds to the condition is performed. If the condition is not met, the operation is not performed and the next condition is evaluated. If one of the specified operations is performed to delete an event, the remaining operations are not performed on the event.
	e_if_else	Performs an operation based on the evaluation result of a condition.
	e_switch	Performs an operation if a condition is met. Multiple condition-operation pairs can be specified. If a condition is met, the operation that corresponds to the condition is performed and a result is returned. If the condition is not met, the operation is not performed and the next condition is evaluated. If no specified conditions are met and the default parameter is specified, the default operations that are specified in the default parameter are performed and a result is returned. If one of the specified operations is performed to delete an event, the remaining operations are not performed on the event.
	e_compose	Combines multiple operations. This function is used to combine multiple operations in the e_if, e_switch, or e_if_else function. The specified operations are performed on an event in sequence and a result is returned. If one of the specified operations is performed to delete the event, the remaining operations are not performed.
Event processing functions	e_drop	Deletes an event if a condition is met.
	e_keep	Retains an event if a condition is met.
	e_split	Splits the value of a specified field into multiple events. You can use JMESPath to extract the value of the field, and then split the event.
	e_output	Writes an event to a specified destination Logstore. You can specify parameters such as topic, source, and tags. The event is deleted after it is written to the destination Logstore.
	e_coutput	Writes an event to a specified destination Logstore. You can specify parameters such as topic, source, and tags. The event is retained after it is written to the destination Logstore. The retained event continues to be transformed.
	e_to_metric	Converts log data to time series data that can be stored in a Metricstore.
Field processing functions	v	Extracts the value of a field from an event. If multiple field names are passed in the function, the value of the first field that exists is returned.
	e_set	Adds a field or specifies a new value for an existing field.
	e_drop_fields	Deletes the log fields that meet a specified condition.
	e_keep_fields	Retains the log fields that meet a specified condition.
	e_pack_fields	Encapsulates specified log fields, and then assigns the log fields as a value to a new field.
	e_rename	Renames the log fields that meet a specified condition.
Value extraction functions	e_regex	Extracts the value of a field in an event by using a regular expression and assigns the value to other fields.
	e_json	Performs operations on JSON objects in a specified field. You can set parameters to expand JSON data, extract JSON data by using JMESPath expressions, and expand extracted JSON data.
	e_kv	Extracts the key-value pairs of multiple source fields and uses the character specified by the quote parameter to enclose the value in each key-value pair.
	e_kv_delimit	Extracts key-value pairs of multiple source fields by using delimiters.
	e_csv	Extracts the value of a specified source field, parses the value into multiple values by using a custom delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is a comma (,).
	e_tsv	Extracts the value of a specified source field, parses the value into multiple values by using a custom delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is \t.
	e_psv	Extracts the value of a specified source field, parses the value into multiple values by using a custom delimiter, and then assigns the parsed values to the predefined fields. The default delimiter is a vertical bar (|).
	e_syslogrfc	Calculates the values of the facility and severity fields and then returns the corresponding level information. The calculation is based on the value of the priority field in an event based on the syslog protocol.
	e_anchor	Extracts the value between the specified start and end positions.
Mapping and enrichment functions	e_dict_map	Maps the value of a specified field to a field based on a dictionary, and then returns the value of the matched field.
	e_table_map	Maps the value of a specified field to a row in a table, and then returns the value of the field in this row.
	e_search_dict_map	Maps a search string to a key in a dictionary, and then returns the value of the matched key.
	e_search_table_map	Maps a search string to a column in a table, and then returns the value of the field in this column.
Value-added content functions	e_threat_intelligence	Obtains threat intelligence from a log field, and then sends the threat intelligence to a specified field.

Expression functions

Type	Function	Description
Event check functions	e_has	Checks whether a field exists.
	e_not_has	Checks whether a field does not exist.
	e_search	Searches for specified events in a simplified method.
	e_match, e_match_all, and e_match_any	Checks whether the value of a field in an event meets the specified condition in an expression.
Operator functions	op_if	Returns an expression based on a specified condition.
	op_ifnull	Returns the value of the first expression whose value is not None.
	op_coalesce	Returns the value of the first expression whose value is not None.
	op_nullif	Returns the value None if the value of Expression 1 is equal to the value of Expression 2. Otherwise, the value of Expression 1 is returned.
	op_and	Invokes the AND operation.
	op_not	Invokes the NOT operation.
	op_or	Invokes the OR operation.
	op_eq	Returns the result that is calculated based on the a==b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
	op_ge	Returns the result that is calculated based on the a>=b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
	op_gt	Returns the result that is calculated based on the a>b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
	op_le	Returns the result that is calculated based on the a<=b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
	op_lt	Returns the result that is calculated based on the a<b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
	op_ne	Returns the result that is calculated based on the a!=b condition. The data types of a and b must be the same. For example, a and b are both strings, numbers, or lists.
	op_len	Calculates the number of characters in a text string. You can use this function in expressions that return strings, tuples, lists, or dictionaries.
	op_in	Checks whether a string, tuple, list, or dictionary contains a specified element.
	op_not_in	Checks whether a string, tuple, list, or dictionary does not contain a specified element.
	op_slice	Truncates a specified string, array, or tuple.
	op_index	Returns the element that corresponds to the index of a specified string, array, or tuple.
	op_add	Calculates the sum of multiple values. The values can be strings or numbers.
	op_max	Returns the maximum value among the values of multiple fields or expressions.
	op_min	Returns the minimum value among the values of multiple fields or expressions.
Conversion functions	ct_int	Converts the value of a field or an expression to an integer.
	ct_float	Converts the value of a field or an expression to a floating-point number.
	ct_str	Converts the value of a field or an expression to a string.
	ct_bool	Converts the value of a field or an expression to a Boolean value.
	ct_chr	Converts the ANSI or Unicode value of a field or an expression to a character.
	ct_ord	Converts the value of a field or an expression to an ANSI or a Unicode value.
	ct_hex	Converts the value of a field or an expression to a hexadecimal number.
	ct_oct	Converts the value of a field or an expression to an octal number.
	ct_bin	Converts the value of a field or an expression to a binary number.
	bin2oct	Converts a binary byte string to an octal string.
	bin2hex	Converts a binary string to a hexadecimal string.
Arithmetic functions	op_abs	Returns the absolute value of a passed value.
	op_div_floor	Returns the integer part of the quotient of two passed values.
	op_div_true	Returns the quotient of two passed values.
	op_pow	Returns a passed value raised to a specified power.
	op_mul	Returns the product of two passed values.
	op_neg	Returns the opposite number of a passed value.
	op_mod	Returns the remainder of a passed value divided by the other passed value.
	op_sub	Returns the difference of two passed values.
	op_round	Returns a passed value rounded.
	op_sum	Returns the sum of passed values.
	mat_ceil	Returns a passed value rounded up to the nearest integer.
	mat_exp	Returns Euler's number raised to the power of a passed value.
	mat_fabs	Returns the absolute value of a passed value.
	mat_floor	Returns a passed value rounded down to the nearest integer.
	mat_log	Returns the logarithm of a passed value with the other passed value as the base.
	mat_log10	Returns the base-10 logarithm of a passed value.
	mat_sqrt	Returns the square root of a passed value.
	mat_degrees	Converts radians to degrees.
	mat_radians	Converts degrees to radians.
	mat_sin	Returns the sine of a passed value (in radians).
	mat_cos	Returns the cosine of a passed value (in radians).
	mat_tan	Returns the tangent of a passed value (in radians).
	mat_acos	Returns the arc cosine (in radians) of a passed value.
	mat_asin	Returns the arc sine (in radians) of a passed value.
	mat_atan	Returns the arc tangent (in radians) of a passed value.
	mat_atan2	Returns the arc tangent of the X-coordinate and the Y-coordinate.
	mat_atanh	Returns the inverse hyperbolic tangent of a passed value.
	mat_hypot	Returns the Euclidean norm of two passed values.
String functions	str_format	Formats a string.
	str_join	Concatenates the elements in a sequence with a specified string to generate a new string.
	str_zip	Splits two value strings or expression strings concurrently, and then merges the strings into a single string.
	str_encode	Encodes a string by using a specified encoding format.
	str_decode	Decodes a string by using a specified decoding format.
	str_hex_escape_encode	Escapes special characters to Chinese characters. Hexadecimal characters are supported.
	str_sort	Sorts the characters in a string.
	str_reverse	Reverses the characters in a string.
	str_replace	Replaces an old string with a new string based on a rule.
	str_logtash_config_normalize	Converts the data format of the Logstash configuration language to the JSON format.
	str_translate	Replaces the specified characters in a string with mapping characters.
	str_strip	Deletes specified characters from a string.
	str_lstrip	Deletes specified characters from the start of a string.
	str_rstrip	Deletes specified characters from the end of a string.
	str_lower	Converts all uppercase characters in a string to lowercase characters.
	str_upper	Converts all lowercase characters in a string to uppercase characters.
	str_title	Capitalizes the first letter of each word in a string, and converts all other letters in the string to lowercase letters.
	str_capitalize	Capitalizes the first letter of a string, and converts all other letters in the string to lowercase letters.
	str_swapcase	Interchanges the uppercase letters and lowercase letters in a string.
	str_count	Counts the number of occurrences of a character in a string.
	str_find	Checks whether an original string contains a specified substring.
	str_rfind	Finds the position of the last occurrence of a character or a substring in a string.
	str_endswith	Checks whether a string ends with a specified suffix.
	str_startswith	Checks whether a string starts with a specified string.
	str_split	Splits a string with a specified delimiter.
	str_splitlines	Splits a string with a line feed.
	str_partition	Splits a string with a specified delimiter into three parts from left to right.
	str_rpartition	Splits a string with a specified delimiter into three parts from right to left.
	str_center	Pads a string to a specified length with a specified character.
	str_ljust	Pads a string to a specified length from the end with a specified character.
	str_rjust	Pads a string to a specified length from the start with a specified character.
	str_zfill	Pads a string to a specified length from the start with 0.
	str_expandtabs	Converts \t in a string to a space character.
	str_isalnum	Checks whether a string consists of only letters and digits.
	str_isalpha	Checks whether a string consists of only letters.
	str_isascii	Checks whether a string is in the ASCII table.
	str_isdecimal	Checks whether a string contains only decimal characters.
	str_isdigit	Checks whether a string consists of only digits.
	str_isidentifier	Checks whether a string is a valid Python identifier. This function can also check whether a variable name is valid.
	str_islower	Checks whether a string consists of lowercase letters.
	str_isnumeric	Checks whether a string consists of digits.
	str_isprintable	Checks whether all characters in a string are printable characters.
	str_isspace	Checks whether a string consists of only space characters.
	str_istitle	Checks whether the first letter of each word in a string is in uppercase and whether all other letters in the string are in lowercase.
	str_isupper	Checks whether all letters in a string are uppercase letters.
	str_uuid	Generates a random universally unique identifier (UUID).
Date and time functions	dt_parse	Converts the value of a time expression to a datetime object.
	dt_str	Converts the value of a time expression to a string.
	dt_parsetimestamp	Converts the value of a time expression to a UNIX timestamp.
	dt_prop	Obtains the specified attribute of the value of a time expression.
	dt_now	Obtains the current datetime.
	dt_today	Obtains the current date and excludes the time.
	dt_utcnow	Obtains the current datetime in the current time zone.
	dt_fromtimestamp	Converts a UNIX timestamp to a datetime object.
	dt_utcfromtimestamp	Converts a UNIX timestamp to a datetime object in the current time zone.
	dt_strptime	Converts a time string to a datetime object.
	dt_currentstamp	Obtains a UNIX timestamp.
	dt_totimestamp	Converts a datetime object to a UNIX timestamp.
	dt_strftime	Converts a datetime object to a string in a specified format.
	dt_strftimestamp	Converts a UNIX timestamp to a string in a specified format.
	dt_truncate	Truncates the value of a time expression based on a specified time granularity.
	dt_add	Modifies the value of a time expression based on a specified time granularity.
	dt_MO	Offsets the specified time to the date of the previous or next Nth Monday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_TU	Offsets the specified time to the date of the previous or next Nth Tuesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_WE	Offsets the specified time to the date of the previous or next Nth Wednesday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_TH	Offsets the specified time to the date of the previous or next Nth Thursday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_FR	Offsets the specified time to the date of the previous or next Nth Friday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_SA	Offsets the specified time to the date of the previous or next Nth Saturday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_SU	Offsets the specified time to the date of the previous or next Nth Sunday. The offset value N is passed to the weekday parameter of the dt_add function, where N can be a positive or negative integer. To pass a negative integer, use op_neg(positive integer).
	dt_astimezone	Converts the value of a time expression to a datetime object in a specified time zone.
	dt_diff	Obtains the difference between the values of two time expressions based on a specified time granularity.
Regular expression functions	regex_select	Extracts a specified value based on a regular expression.
	regex_findall	Obtains all values that match a regular expression.
	regex_match	Checks whether a value matches a regular expression.
	regex_replace	Replaces specified characters in a string based on a regular expression.
	regex_split	Splits a value into multiple values.
Grok function	grok	Extracts a specified value based on a regular expression.
Structured data functions	json_select	Extracts or calculates a specific value from a JSON expression based on the JMESPath syntax.
	json_parse	Converts a specified value to a JSON object.
	xml_to_json	Converts XML-formatted data to JSON-formatted data.
	geo_parse	Identifies the city, province, and country based on an IP address.
	gzip_compress	Compresses specified data, encodes the compressed data by using the Base64 algorithm, and then returns the transformed data.
	gzip_decompress	Decodes data by using the Base64 algorithm, decompresses the decoded data, and then returns the transformed data.
	ip_cidrmatch	Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.
	ip_version	Checks whether the version of an IP address is IPv4 or IPv6.
	ip_type	Identifies the type of an IP address.
	ip_makenet	Converts an IP address to a CIDR block.
	ip_to_format	Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.
	ip_overlaps	Checks whether two CIDR blocks overlap.
Encoding and decoding functions	url_encoding	Encodes URL data.
	url_decoding	Decodes URL data.
	base64_encoding	Encodes data by using the Base64 algorithm.
	base64_decoding	Decodes data by using the Base64 algorithm.
	html_encoding	Encodes data in the HTML format.
	html_decoding	Decodes HTML-encoded data.
	md5_encoding	Encodes data by using the MD5 algorithm.
	sha1_encoding	Encodes data by using the SHA1 algorithm.
	ip2long	Converts an IP address to a value of the long type.
	long2ip	Converts a value of the long type to an IP address.
Parsing functions	ua_parse_device	Parses the device information in a User-Agent HTTP header.
	ua_parse_os	Parses the operating system information in a User-Agent HTTP header.
	ua_parse_agent	Parses the browser information in a User-Agent HTTP header.
	ua_parse_all	Parses all information in a User-Agent HTTP header.
List functions	lst_make	Constructs a list.
	lst_insert	Inserts elements to a specified position in a list.
	lst_append	Adds elements to the end of a list.
	lst_delete_at	Deletes the element at a specified position from a list.
	lst_reverse	Reverses the order of elements in a list.
	lst_get	Obtains an element from a list or tuple.
Dictionary functions	dct_make	Constructs a dictionary.
	dct_update	Updates a dictionary.
	dct_delete	Deletes key-value pairs from a dictionary.
	dct_keys	Obtains the keys of a dictionary.
	dct_values	Obtains the values of a dictionary.
	dct_get	Obtains the value that corresponds to a specified key in a dictionary.
Table functions	tab_parse_csv	Constructs a table from a comma-separated values (CSV) file.
	tab_to_dict	Constructs a dictionary from a table.
Resource functions	res_local	Obtains the values of advanced parameters of a data transformation task.
	res_rds_mysql	Obtains data from a specified database table in ApsaraDB RDS for MySQL. The data can be refreshed at regular intervals.
	res_log_logstore_pull	Pulls data from another Logstore when a Logstore is being transformed. You can pull data and maintain data in a table in a continuous manner.
	res_oss_file	Obtains the content of an object in a specified bucket from Object Storage Service (OSS). The object can be refreshed at regular intervals.