This topic describes the expression functions that you can use to transform data in Simple Log Service.
Category | Function | Description |
Event check functions | e_has | Checks whether a field exists. |
e_not_has | Checks whether a field does not exist. | |
e_search | Searches for a log by using simplified, Lucene-like query syntax. | |
e_match, e_match_all, and e_match_any | Checks whether the value of a field in a log matches a specified expression. | |
Operator functions | op_if | Returns the value of an expression based on a specified condition. |
op_ifnull | Returns the value of the first expression whose value is not None. | |
op_coalesce | Returns the value of the first expression whose value is not None. | |
op_nullif | Returns the value None if the value of Expression 1 is equal to the value of Expression 2. If the values of Expression 1 and Expression 2 are different, the value of Expression 1 is returned. | |
op_and | Returns a Boolean result after two or more specified fields are evaluated by using the logical AND operator. Each field can be of an arbitrary data type. | |
op_not | Returns a reverse Boolean result after two or more specified fields are evaluated by using the logical NOT operator. Each field can be of an arbitrary data type. | |
op_or | Returns a Boolean result after two or more specified fields are evaluated by using the logical OR operator. Each field can be of an arbitrary data type. If one or more of the specified fields are evaluated to true, the value True is returned. Otherwise, the value False is returned. | |
op_eq | Returns True or False based on the a==b condition. | |
op_ge | Returns True or False based on the a>=b condition. | |
op_gt | Returns True or False based on the a>b condition. | |
op_le | Returns True or False based on the a<=b condition. | |
op_lt | Returns True or False based on the a<b condition. | |
op_ne | Returns True or False based on the a!=b condition. | |
op_len | Calculates the number of characters in a text string. This function applies to strings or expressions that return tuples, lists, or dictionaries. | |
op_in | Checks whether a string, tuple, list, or dictionary contains a specified element. This function returns True or False. | |
op_not_in | Checks whether a string, tuple, list, or dictionary does not contain a specified element. This function returns True or False. | |
op_slice | Extracts strings from a specified string, array, or tuple. | |
op_index | Returns the element that corresponds to the index of a specified string, array, or tuple. | |
op_add | Calculates the sum of multiple values. The values can be strings or numbers. | |
op_max | Returns the largest value among the values of multiple fields or expressions. | |
op_min | Returns the smallest value among the values of multiple fields or expressions. | |
Conversion functions | ct_int | Converts the value of a field or an expression to an integer. |
ct_float | Converts the value of a field or an expression to a floating-point number. | |
ct_str | Converts the value of a field or an expression to a string. | |
ct_bool | Converts the value of a field or an expression to a Boolean value. | |
ct_chr | Converts the ANSI or Unicode value of a field or an expression to a character. | |
ct_ord | Converts the value of a field or an expression to an ANSI value or a Unicode value. | |
ct_hex | Converts the value of a field or an expression to a hexadecimal number. | |
ct_oct | Converts the value of a field or an expression to an octal number. | |
ct_bin | Converts the value of a field or an expression to a binary number. | |
bin2oct | Converts a binary byte string to an octal string. | |
bin2hex | Converts a binary byte string to a hexadecimal string. | |
Arithmetic functions | op_abs | Returns the absolute value of an input value. |
op_div_floor | Returns the integer part of the quotient of two input values. | |
op_div_true | Returns the quotient of two input values. | |
op_pow | Raises an input value to a specified power. | |
op_mul | Returns the product of two input values. | |
op_neg | Returns the opposite number of an input value. | |
op_mod | Returns the remainder of an input value divided by the other input value. | |
op_sub | Returns the difference between two input values. | |
op_round | Rounds an input value. | |
op_sum | Returns the sum of input values. | |
mat_ceil | Rounds an input value up to the nearest integer. | |
mat_exp | Raises e to a specified power. | |
mat_fabs | Returns the absolute value of an input value. | |
mat_floor | Rounds an input value down to the nearest integer. | |
mat_log | Returns the logarithm of an input value with the other input value as the base. | |
mat_log10 | Returns the base-10 logarithm of an input value. | |
mat_sqrt | Returns the square root of an input value. | |
mat_degrees | Converts radians to degrees. | |
mat_radians | Converts degrees to radians. | |
mat_sin | Returns the sine of an input value in radians. | |
mat_cos | Returns the cosine of an input value in radians. | |
mat_tan | Returns the tangent of an input value in radians. | |
mat_acos | Returns the arc cosine of an input value in radians. | |
mat_asin | Returns the arc sine of an input value in radians. | |
mat_atan | Returns the arc tangent of an input value in radians. | |
mat_atan2 | Returns the arc tangent of X and Y coordinates. | |
mat_atanh | Returns the inverse hyperbolic tangent of an input value. | |
mat_hypot | Returns the Euclidean norm of two input values. | |
String functions | str_format | Formats strings in a specified format. |
str_join | Concatenates input strings by using a specified connector to generate a new string. | |
str_zip | Concurrently splits two values or strings that are returned by expressions and combines the results into one string. | |
str_encode | Encodes a string by using a specified encoding format. | |
str_decode | Decodes an input value by using a specified encoding format. | |
str_hex_escape_encode | Escapes special characters. The function can escape hexadecimal characters to Chinese characters. | |
str_sort | Sorts a specified object. | |
str_reverse | Reverses a string. | |
str_replace | Replaces an existing string with a specified string based on a specified rule. | |
str_logtash_config_normalize | Converts the data in the Logstash configuration language to the JSON format. | |
str_translate | Replaces specified characters in a string with mapping characters. | |
str_strip | Deletes specified characters from a string. | |
str_lstrip | Deletes specified characters from the start of a string. | |
str_rstrip | Deletes specified characters from the end of a string. | |
str_lower | Converts all uppercase letters in a string to lowercase letters. | |
str_upper | Converts all lowercase letters in a string to uppercase letters. | |
str_title | Capitalizes the first letter of each word in a string and converts the other letters in the string to lowercase letters. | |
str_capitalize | Capitalizes the first letter of a string and converts the other letters in the string to lowercase letters. | |
str_swapcase | Converts the uppercase letters in a string to lowercase letters and lowercase letters to uppercase letters. | |
str_count | Counts the number of occurrences of a character in a string. | |
str_find | Checks whether a string contains a specified substring. | |
str_rfind | Returns the position of the last occurrence of a specified character in a string. | |
str_endswith | Checks whether a string ends with a specified suffix. | |
str_startswith | Checks whether a string starts with a specified string. | |
str_split | Splits a string by using a specified delimiter. | |
str_splitlines | Splits a string by using a line feed. | |
str_partition | Splits a string into three parts from left to right by using a specified delimiter. | |
str_rpartition | Splits a string into three parts from right to left by using a specified delimiter. | |
str_center | Pads a string to a specified length by using a specified character. | |
str_ljust | Right pads a string to a specified length by using a specified character. | |
str_rjust | Left pads a string to a specified length by using a specified character. | |
str_zfill | Left pads a string to a specified length by using 0. | |
str_expandtabs | Converts \t in a string to spaces. | |
str_isalnum | Checks whether a string contains only letters and digits. | |
str_isalpha | Checks whether a string contains only letters. | |
str_isascii | Checks whether a string is in the ASCII table. | |
str_isdecimal | Checks whether a string contains only decimal characters. | |
str_isdigit | Checks whether a string contains only digits. | |
str_isidentifier | Checks whether a string is a valid Python identifier or checks whether a variable name is valid. | |
str_islower | Checks whether a string contains lowercase letters. | |
str_isnumeric | Checks whether a string contains digits. | |
str_isprintable | Checks whether all characters in a string are printable characters. | |
str_isspace | Checks whether a string contains only spaces. | |
str_istitle | Checks whether the first letter of each word in a string is in uppercase and the other letters in the string are in lowercase. | |
str_isupper | Checks whether all letters in a string are in uppercase. | |
str_uuid | Generates a random universally unique identifier (UUID). | |
Date and time functions | dt_parse | Converts a value or the value of a time expression to a datetime object. |
dt_str | Converts a value or the value of a time expression to a string. | |
dt_parsetimestamp | Converts a value or the value of a time expression to a UNIX timestamp. | |
dt_prop | Obtains a specific attribute from the value of a time expression, such as the day or year. | |
dt_now | Obtains the current date and time. | |
dt_today | Obtains only the current date. | |
dt_utcnow | Obtains the current date and time from the current time zone. | |
dt_fromtimestamp | Converts a UNIX timestamp to a datetime object. | |
dt_utcfromtimestamp | Converts a UNIX timestamp to a datetime object in the current time zone. | |
dt_strptime | Parses a time string into a datetime object. | |
dt_currentstamp | Obtains the current UNIX timestamp. | |
dt_totimestamp | Converts a datetime object to a UNIX timestamp. | |
dt_strftime | Converts a datetime object to a string in a specified format. | |
dt_strftimestamp | Converts a UNIX timestamp to a string in a specified format. | |
dt_truncate | Extracts a time value from a value or the value of a time expression based on a specified time granularity. | |
dt_add | Changes a value or the value of a time expression based on a specified time granularity. | |
dt_MO | Offsets a specified time to the date of the previous or following Nth Monday. The offset value N is passed to the weekday parameter of the dt_add function. | |
dt_TU | Offsets a specified time to the date of the previous or following Nth Tuesday. The offset value N is passed to the weekday parameter of the dt_add function. | |
dt_WE | Offsets a specified time to the date of the previous or following Nth Wednesday. The offset value N is passed to the weekday parameter of the dt_add function. | |
dt_TH | Offsets a specified time to the date of the previous or following Nth Thursday. The offset value N is passed to the weekday parameter of the dt_add function. | |
dt_FR | Offsets a specified time to the date of the previous or following Nth Friday. The offset value N is passed to the weekday parameter of the dt_add function. | |
dt_SA | Offsets a specified time to the date of the previous or following Nth Saturday. The offset value N is passed to the weekday parameter of the dt_add function. | |
dt_SU | Offsets a specified time to the date of the previous or following Nth Sunday. The offset value N is passed to the weekday parameter of the dt_add function. | |
dt_astimezone | Converts a value or the value of a time expression to a datetime object in a specified time zone. | |
dt_diff | Returns the difference between two values or between the values of two time expressions based on a specified time granularity. | |
Regular expression functions | regex_select | Extracts a value that matches a regular expression. |
regex_findall | Extracts all values that match a regular expression. | |
regex_match | Checks whether a value matches a regular expression. | |
regex_replace | Replaces the characters that match a regular expression in a string. | |
regex_split | Splits a string into an array of strings. | |
Grok function | grok | Extracts a value that matches a regular expression. |
Structured data functions | json_select | Extracts or calculates a specified value from a JSON expression based on the JMESPath syntax. |
json_parse | Parses a specified value into a JSON object. | |
xml_to_json | Converts XML data to JSON data, and then expands the converted data. | |
IP address parsing functions | geo_parse | Parses an IP address into the information about the city, province, and country of the IP address. |
ip_cidrmatch | Checks whether an IP address belongs to a CIDR block. | |
ip_version | Checks whether the version of an IP address is IPv4 or IPv6. | |
ip_type | Identifies the type of an IP address and checks whether the type of the IP address is private or public. | |
ip_makenet | Converts an IP address to a CIDR block. | |
ip_to_format | Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block. | |
ip_overlaps | Checks whether two CIDR blocks overlap. | |
ip2long | Converts an IP address to a value of the long type. | |
long2ip | Converts a value of the long type to an IP address. | |
Encoding and decoding functions | url_encoding | Performs URL encoding on data. |
url_decoding | Performs URL decoding on data. | |
protobuf_decoding | Parses data into the JSON format by using a specified Protobuf template. | |
str_encode | Encodes a string by using a specified encoding format. | |
str_decode | Decodes an input value by using a specified encoding format. | |
base64_encoding | Encodes data by using the Base64 algorithm. | |
base64_decoding | Decodes data by using the Base64 algorithm. | |
html_encoding | Encodes data in the HTML format. | |
html_decoding | Decodes HTML-encoded data. | |
md5_encoding | Encodes data by using the MD5 algorithm. | |
sha1_encoding | Encodes data by using the SHA1 algorithm. | |
crc32_encoding | Calculates a cyclic redundancy check (CRC) code for data. | |
gzip_compress | Compresses and encodes data. | |
gzip_decompress | Decompresses compressed data. | |
zlib_compress | Compresses and encodes data. | |
zlib_decompress | Decompresses compressed data. | |
aes_encrypt | Encrypts data by using the AES algorithm. | |
aes_decrypt | Decrypts data by using the AES algorithm. | |
jwt_encoding | Encodes JSON data based on the JSON Web Token (JWT) standard. | |
jwt_decoding | Decodes data to raw JSON data based on the JWT standard. | |
hashids_encoding | Encodes data by using the Hashids library. | |
hashids_decoding | Decodes data that is encoded by using the Hashids library. | |
Parsing functions | ua_parse_device | Parses User-Agent and returns the device information. |
ua_parse_os | Parses User-Agent and returns the operating system information. | |
ua_parse_agent | Parses User-Agent and returns the browser information. | |
ua_parse_all | Parses User-Agent and returns all information. | |
url_parse | Parses a URL and returns the components of the URL. | |
url_parse_qs | Parses the query string of a URL and returns the components of the query string. | |
List functions | lst_make | Constructs a list. |
lst_insert | Inserts elements to a specified position in a list. | |
lst_append | Appends elements to a list. | |
lst_delete_at | Deletes the element at a specified position from a list. | |
lst_reverse | Reverses the order of elements in a list. | |
lst_get | Obtains an element from a list or tuple. | |
Dictionary functions | dct_make | Constructs a dictionary. |
dct_update | Updates a dictionary. | |
dct_delete | Deletes key-value pairs from a dictionary. | |
dct_keys | Obtains the keys of a dictionary. | |
dct_values | Obtains the values of a dictionary. | |
dct_get | Obtains the value that corresponds to a specified key in a dictionary. | |
Table functions | tab_parse_csv | Constructs a table from comma-separated values (CSV) text. |
tab_to_dict | Constructs a dictionary from a table. | |
Resource functions | res_local | Pulls the values of advanced parameters from the current data transformation task. |
res_rds_mysql | Pulls data from a specified table in a database that is created on an ApsaraDB RDS for MySQL instance or obtains the execution result of an SQL statement. The data can be updated at regular intervals. | |
res_log_logstore_pull | Pulls data from another Logstore when you transform data in a Logstore. You can continuously pull data. | |
res_oss_file | Pulls data from an object in a specified Object Storage Service (OSS) bucket. The data can be updated at regular intervals. |