Log Transformation Functions Overview - Simple Log Service

This topic lists all the functions for data transformation tasks.

Global processing functions

Function Classification	Function Name	Description
Flow control functions	e_if	Performs an operation if a specified condition is met. You can specify multiple condition-operation pairs. If a condition is met, the function performs the operation that corresponds to the condition. If the condition is not met, the function does not perform the operation, but evaluates the next condition. If the function performs an operation that deletes a log, the function no longer performs other operations on the log.
	e_if_else	Performs an operation based on the evaluation result of a condition.
	e_switch	A combination of conditions and operations. If a condition is met, the function performs the operation that corresponds to the condition and returns the result. If the condition is not met, the function does not perform the operation, but evaluates the next condition. If no specified conditions are met and the default parameter is configured, the function performs the operation that is specified by the default parameter and returns the result. If an operation deletes a log, no subsequent operations are executed.
	e_compose	Combines multiple operations. The function is commonly used in the e_if, e_switch, or e_if_else function. The function performs specified operations on a log in sequence and returns the result. If the function performs an operation that deletes a log, the function no longer performs other operations on the log.
Event operation functions	e_drop	Discards a log if a specified condition is met.
	e_keep	Retains a log if a specified condition is met.
	e_split	Splits a log into multiple logs based on the value of a specified field. You can also use the JMESPath expression to extract the value of the field, and then split the log.
	e_output	Outputs logs to a specified Logstore. Configure the topic, source, tag, and shard hash information for output. After output, subsequent transformation rules are not executed for the log.
	e_coutput	Outputs logs to a specified Logstore. Configure the topic, source, tag, and shard hash information for output. After output, subsequent transformation rules continue to be executed.
	e_to_metric	Converts logs to metrics that can be stored in a Metricstore.
Field operation functions	v	Extracts the value of a field from a log. If you specify the names of multiple fields for the function, the function returns the value of the first field that exists in the log.
	e_set	Adds a field or specifies a new value for an existing field.
	e_drop_fields	Deletes the log fields that meet a specified condition.
	e_keep_fields	Retains the log fields that meet a specified condition.
	e_pack_fields	Encapsulates log fields and assigns the log fields as a value to a new field.
	e_rename	Renames the log fields that meet a specified condition.
Value extraction functions	e_regex	Extracts the value of a field based on a regular expression and assigns the value to other fields.
	e_json	Manages JSON objects in a specified field in a log. You can configure the parameters to expand JSON data, extract JSON data using the JMESPath expression, or expand the extracted JSON data.
	e_kv	Extracts key-value pairs from multiple input fields using a specified quote.
	e_kv_delimit	Extracts key-value pairs from input fields using a specified delimiter.
	e_csv	Extracts multiple fields from a specified field using a specified delimiter and predefined field names. The default delimiter is a comma (,).
	e_tsv	Extracts multiple fields from a specific field using a custom delimiter and predefined field names. The default delimiter is `\t`.
	e_psv	Extracts multiple fields from a specified source field using a specified delimiter and predefined field names. The default delimiter is a vertical bar (\|).
	e_syslogrfc	According to the Syslog protocol, the facility and severity values are derived from the priority value, and the corresponding log level is determined.
	e_anchor	Extracts strings using the rules specified by anchor_rules.
Mapping and enrichment functions	e_dict_map	The field mapping dictionary maps an input field to a new field.
	e_table_map	You can look up the corresponding row in the mapping table and return the field value based on the input field.
	e_tablestore_map	Enriches a raw log using a data table in Tablestore as the dimension table.
	e_redis_map	Enriches a raw log using a data table in ApsaraDB for Redis as the dimension table.

Expression functions

Function classification	Function name	Description
Event check functions	e_has	Checks whether a field exists.
	e_not_has	Checks whether a field does not exist.
	e_search	Searches for a log using a query syntax that is similar to Lucene.
	e_match, e_match_all, and e_match_any	Checks whether the value of a log field meets the conditions specified in a regular expression.
Operator functions	op_if	Returns the value of an expression based on a specified condition.
	op_ifnull	Returns the value of the first expression whose value is not None.
	op_coalesce	Returns the value of the first expression whose value is not None.
	op_nullif	Returns none if the value of Expression 1 is equal to the value of Expression 2. If the values of Expression 1 and Expression 2 are different, the value of Expression 1 is returned.
	op_and	Evaluates the specified expressions using the logical AND operator and returns True if all specified expressions evaluate to true. The value of each expression can be of an arbitrary data type.
	op_not	Evaluates a specified expression using the logical NOT operator and returns the inverse Boolean value of the specified expression. The value of the expression can be of an arbitrary data type.
	op_or	Evaluates the specified expressions using the logical OR operator, and returns True if a specified expression evaluates to true or returns False if all specified expressions evaluate to false. The value of each expression can be of an arbitrary data type.
	op_eq	Calculates based on the `a==b` condition, returning True or False.
	op_ge	Calculates based on the `a>=b` condition, returning True or False.
	op_gt	Calculates based on the `a>b` condition, returning True or False.
	op_le	Calculates based on the `a<=b` condition, returning True or False.
	op_lt	Calculates based on the `a<b` condition, returning True or False.
	op_ne	Calculates based on the `a!=b` condition, returning True or False.
	op_len	Calculates the number of characters in a text string. This function applies to strings or expressions that return tuples, lists, or dictionaries.
	op_in	Checks whether a string, tuple, list, or dictionary contains a specified element and returns True or False.
	op_not_in	Checks whether a string, tuple, list, or dictionary does not contain a specified element and returns True or False.
	op_slice	Truncates the specified string, array, or tuple.
	op_index	Returns the element that corresponds to the index of a specified string, array, or tuple.
	op_add	Calculates the sum of multiple values. The values can be strings or numbers.
	op_max	Returns the largest value among the values of multiple fields or expressions.
	op_min	Returns the smallest value among the values of multiple fields or expressions.
Conversion functions	ct_int	Converts the value of a field or an expression to an integer.
	ct_float	Converts the value of a field or an expression to a floating-point number.
	ct_str	Converts the value of a field or an expression to a string.
	ct_bool	Converts the value of a field or an expression to a Boolean value.
	ct_chr	Converts the ANSI or Unicode value of a field or an expression to a character.
	ct_ord	Converts the characters in a field or expression to their corresponding ANSI or Unicode values.
	ct_hex	Converts the value of a field or an expression to a hexadecimal number.
	ct_oct	Converts the value of a field or an expression to an octal number.
	ct_bin	Converts the value of a field or an expression to a binary number.
	bin2oct	Converts a binary byte string to an octal string.
	bin2hex	Converts a binary byte string to a hexadecimal string.
Arithmetic functions	op_abs	Returns the absolute value of an input value.
	op_div_floor	Calculates the integer division of the input value.
	op_div_true	Divides the input value.
	op_pow	Returns an input value raised to a specified power.
	op_mul	Multiplies an input value.
	op_neg	Returns the opposite number of an input value.
	op_mod	Performs a modulo calculation on the input value.
	op_sub	Applies a subtraction to the input value.
	op_round	Rounds the incoming value.
	op_sum	Returns the sum of input values.
	mat_ceil	Rounds an input value rounded up to the nearest integer.
	mat_exp	Represents the exponential function with base e.
	mat_fabs	Returns the absolute value of an input value.
	mat_floor	Rounds an input value down to the nearest integer.
	mat_log	Calculates the logarithm of the input value.
	mat_log10	Returns the base-10 logarithm of an input value.
	mat_sqrt	Returns the square root of an input value.
	mat_degrees	Converts radians to degrees.
	mat_radians	Converts degrees to radians.
	mat_sin	Returns the sine of an input value in radians.
	mat_cos	Returns the cosine of an input value in radians.
	mat_tan	Returns the tangent of an input value in radians.
	mat_acos	Returns the arc cosine of an input value in radians.
	mat_asin	Returns the arc sine of an input value in radians.
	mat_atan	Returns the arc tangent of an input value in radians.
	mat_atan2	Returns the arc tangent of X and Y coordinates.
	mat_atanh	Returns the inverse hyperbolic tangent of an input value.
	mat_hypot	Compute the Euclidean norm of the input value.
String functions	str_format	You can format the string using the specified format.
	str_join	Concatenates input strings to generate a new string using a specified connector.
	str_zip	Concurrently splits two values or strings that are returned by expressions and combines the results into one string.
	str_encode	Encodes a string using a specified encoding format.
	str_decode	Decodes an input value using a specified encoding format.
	str_hex_escape_encode	Escapes special characters. The function can escape hexadecimal characters to Chinese characters.
	str_sort	Sorts a specified object.
	str_reverse	Reverses a string.
	str_replace	Replaces an existing string with a specified string based on a specified rule.
	str_logtash_config_normalize	Converts data in the Logstash configuration language to the JSON format.
	str_translate	Replaces specified characters in a string with mapping characters.
	str_strip	Deletes specified characters from a string.
	str_lstrip	Deletes specified characters from the start of a string.
	str_rstrip	Deletes specified characters from the end of a string.
	str_lower	Converts all uppercase letters in a string to lowercase letters.
	str_upper	Converts all lowercase letters in a string to uppercase letters.
	str_title	Capitalizes the first letter of each word in a string and converts the other letters in the string to lowercase letters.
	str_capitalize	Capitalizes the first letter of a string and converts the other letters in the string to lowercase letters.
	str_swapcase	Converts the uppercase letters in a string to lowercase letters and lowercase letters to uppercase letters.
	str_count	Counts the number of occurrences of a character in a string.
	str_find	Checks whether a string contains a specified substring.
	str_rfind	Returns the position of the last occurrence of a specified character in a string.
	str_endswith	Checks whether a string ends with a specified suffix.
	str_startswith	Checks whether a string starts with a specified string.
	str_split	Splits a string using a specified delimiter.
	str_splitlines	Splits a string using a line feed.
	str_partition	Splits a string into three parts from left to right using a specified delimiter.
	str_rpartition	Splits a string into three parts from right to left using a specified delimiter.
	str_center	Pads a string to a specified length using a specified character.
	str_ljust	Pads a string to a specified length using a specified character from the end of the string.
	str_rjust	Pads a string to a specified length using a specified character from the start of the string.
	str_zfill	Pads a string to a specified length using 0 from the start of the string.
	str_expandtabs	Converts `\t` characters in a string to spaces.
	str_isalnum	Checks whether a string contains only letters and digits.
	str_isalpha	Checks whether a string contains only letters.
	str_isascii	Checks whether a string is in the ASCII table.
	str_isdecimal	Checks whether a string contains only decimal characters.
	str_isdigit	Checks whether a string contains only digits.
	str_isidentifier	Checks whether a string is a valid Python identifier or checks whether a variable name is valid.
	str_islower	Determines whether a string contains only lowercase letters.
	str_isnumeric	Checks if the string contains only digits.
	str_isprintable	Checks whether all characters in a string are printable characters.
	str_isspace	Checks whether a string contains only spaces.
	str_istitle	Checks whether the first letter of each word in a string is in uppercase and the other letters in the string are in lowercase.
	str_isupper	Checks whether all letters in a string are in uppercase.
	str_uuid	Generates a random universally unique identifier (UUID).
Date and time functions	dt_parse	Converts a value or the value of a time expression to a datetime object.
	dt_str	Converts a value or the value of a time expression to a string.
	dt_parsetimestamp	Converts a value or the value of a time expression to a UNIX timestamp.
	dt_prop	Returns a specific attribute of a value, or returns a specific attribute of the value of a time expression. The attribute can be day or year.
	dt_now	Returns the current date and time.
	dt_today	Return only the current date.
	dt_utcnow	Returns the current datetime object in the current time zone.
	dt_fromtimestamp	Converts a UNIX timestamp to a datetime object.
	dt_utcfromtimestamp	Converts a UNIX timestamp to a datetime object in the current time zone.
	dt_strptime	Parses a time string into a datetime object.
	dt_currentstamp	Returns the current UNIX timestamp.
	dt_totimestamp	Converts a datetime object to a UNIX timestamp.
	dt_strftime	Converts a datetime object to a string in a specified format.
	dt_strftimestamp	Converts a UNIX timestamp to a string in a specified format.
	dt_truncate	Extracts a time value from a value or the value of a time expression based on a specified time granularity.
	dt_add	Changes a value or the value of a time expression based on a specified time granularity.
	dt_MO	The value passed to the `weekday` parameter in the `dt_add` function, representing the offset for a specific Monday.
	dt_TU	The value passed to the `weekday` parameter in the `dt_add` function, representing the offset for a specific Tuesday.
	dt_WE	The value passed to the `weekday` parameter in the `dt_add` function, representing the offset for a specific Wednesday.
	dt_TH	The value passed to the `weekday` parameter in the `dt_add` function, representing the offset for a specific Thursday.
	dt_FR	The value passed to the `weekday` parameter in the `dt_add` function, representing the offset for a specific Friday.
	dt_SA	The value passed to the `weekday` parameter in the `dt_add` function, representing the offset for a specific Saturday.
	dt_SU	The value passed to the `weekday` parameter in the `dt_add` function, representing the offset for a specific Sunday.
	dt_astimezone	Converts a value or the value of a time expression to a datetime object in a specified time zone.
	dt_diff	Returns the difference between two values or between the values of two time expressions based on a specified time granularity.
Regular expression functions	regex_select	Extracts a value that matches a regular expression.
	regex_findall	Extracts all values that match a regular expression.
	regex_match	Checks whether a value matches a regular expression.
	regex_replace	Replaces the characters that match a regular expression in a string.
	regex_split	Splits a string into an array of strings.
Grok function	grok	Extracts a value that matches a regular expression.
Specific structured data functions	json_select	Extracts or calculates specific values from a JSON expression using JMESPath.
	json_parse	Parses a value into a JSON object.
	xml_to_json	Converts XML data to JSON data, and then expands the converted data.
IP address parsing functions	geo_parse	Identifies the city, province, and country based on an IP address.
	ip_cidrmatch	Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.
	ip_version	Checks whether the version of an IP address is IPv4 or IPv6.
	ip_type	Identifies the type of an IP address and checks whether the type of the IP address is private or public.
	ip_makenet	Converts an IP address to a CIDR block.
	ip_to_format	Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.
	ip_overlaps	Checks whether two CIDR blocks overlap.
	ip2long	Converts an IP address to a value of the long type.
	long2ip	Converts a value of the long type to an IP address.
Encoding and decoding functions	url_encoding	Performs URL encoding on data.
	url_decoding	Performs URL decoding on data.
	protobuf_decoding	Parses data into the JSON format using a specified Protobuf template.
	str_encode	Encodes a string using a specified encoding format.
	str_decode	Decodes an input value using a specified encoding format.
	base64_encoding	Encodes data using the Base64 algorithm.
	base64_decoding	Decodes data using the Base64 algorithm.
	html_encoding	Encodes data in the HTML format.
	html_decoding	Decodes HTML-encoded data.
	md5_encoding	Encodes data using the MD5 algorithm.
	sha1_encoding	Encodes data using the SHA1 algorithm.
	crc32_encoding	Calculates a cyclic redundancy check (CRC) code for data.
	gzip_compress	Compresses and encodes data.
	gzip_decompress	Decompresses compressed data.
	zlib_compress	Compresses and encodes data.
	zlib_decompress	Decompresses compressed data.
	aes_encrypt	Encrypts data using the AES algorithm.
	aes_decrypt	Decrypts data using the AES algorithm.
	jwt_encoding	Encodes JSON data based on the JSON Web Token (JWT) standard.
	jwt_decoding	Decodes data to raw JSON data based on the JWT standard.
	hashids_encoding	Encodes data using the Hashids library.
	hashids_decoding	Decodes data that is encoded using the Hashids library.
Parsing functions	ua_parse_device	Parses User-Agent and returns the device information.
	ua_parse_os	Parses User-Agent and returns the operating system information.
	ua_parse_agent	Parses User-Agent and returns the browser information.
	ua_parse_all	Parses User-Agent and returns all information.
	url_parse	Parses a URL and returns the components of the URL.
	url_parse_qs	Parses the query string of a URL and returns the components of the query string.
List functions	lst_make	Constructs a list.
	lst_insert	Inserts elements to a specified position in a list.
	lst_append	Appends elements to a list.
	lst_delete_at	Deletes the element at a specified position from a list.
	lst_reverse	Reverses the order of elements in a list.
	lst_get	Returns the element at a specified position in a list or a tuple.
Dictionary functions	dct_make	Constructs a dictionary.
	dct_update	Updates a dictionary.
	dct_delete	Deletes key-value pairs from a dictionary.
	dct_keys	Returns the keys of a dictionary.
	dct_values	Returns the values of a dictionary.
	dct_get	Returns the value that corresponds to a specified key in a dictionary.
Table functions	tab_parse_csv	Constructs a table from comma-separated values (CSV) text.
Table functions	tab_to_dict	Constructs a dictionary from a table.
Resource functions	res_local	Pulls the values of advanced parameters from the current data transformation job.
	res_rds_mysql	Pulls data from a specified table in a database that is created on an ApsaraDB RDS for MySQL instance or obtains the execution result of an SQL statement. The data and result can be updated at regular intervals.
	res_log_LogStore_pull	Pulls data from another Logstore. Supports continuous data pulling.
	res_oss_file	Pulls data from an object in a specified Object Storage Service (OSS) bucket. The data can be updated at regular intervals.