All Products
Search
Document Center

Simple Log Service:Function overview

Last Updated:Jan 03, 2024

This topic describes the functions that you can call to transform data in Simple Log Service.

Global processing functions

Category

Function

Description

Flow control functions

e_if

Performs an operation if a specified condition is met. You can specify multiple condition-operation pairs.

  • If a condition is met, the function performs the operation that corresponds to the condition. If the condition is not met, the function does not perform the operation, but evaluates the next condition.

  • If the function performs an operation that deletes a log, the function no longer performs other operations on the log.

e_if_else

Performs an operation based on the evaluation result of a condition.

e_switch

Performs an operation if a specified condition is met. You can specify multiple condition-operation pairs.

  • If a condition is met, the function performs the operation that corresponds to the condition and returns the result. If the condition is not met, the function does not perform the operation, but evaluates the next condition.

  • If no specified conditions are met and the default parameter is configured, the function performs the operation that is specified by the default parameter and returns the result.

  • If the function performs an operation that deletes a log, the function no longer performs other operations on the log.

e_compose

Combines multiple operations.

  • The function is commonly used in the e_if, e_switch, or e_if_else function.

  • The function performs specified operations on a log in sequence and returns the result.

  • If the function performs an operation that deletes a log, the function no longer performs other operations on the log.

Event processing functions

e_drop

Discards a log if a specified condition is met.

e_keep

Retains a log if a specified condition is met.

e_split

Splits a log into multiple logs based on the value of a specified field. You can also use the JMESPath expression to extract the value of the field, and then split the log.

e_output

Writes a log to a specified Logstore. You can specify the topic, source, tags and MD5 hash keys of a shard for the log. The log is deleted after it is written to the specified Logstore. The system no longer transforms the log.

e_coutput

Writes a log to a specified Logstore. You can specify the topic, source, tags and MD5 hash keys of a shard for the log. The log is retained after it is written to the specified Logstore. The system continues to transform the retained log.

e_to_metric

Converts logs to metrics that can be stored in a Metricstore.

Field processing functions

v

Extracts the value of a field from a log. If you specify the names of multiple fields for the function, the function returns the value of the first field that exists in the log.

e_set

Adds a field or specifies a new value for an existing field.

e_drop_fields

Deletes the log fields that meet a specified condition.

e_keep_fields

Retains the log fields that meet a specified condition.

e_pack_fields

Encapsulates log fields and assigns the log fields as a value to a new field.

e_rename

Renames the log fields that meet a specified condition.

Value extraction functions

e_regex

Extracts the value of a field based on a regular expression and assigns the value to other fields.

e_json

Manages JSON objects in a specified field in a log. You can configure the parameters to expand JSON data, extract JSON data by using the JMESPath expression, or expand the extracted JSON data.

e_kv

Extracts key-value pairs from multiple input fields by using a specified quote.

e_kv_delimit

Extracts key-value pairs from input fields by using a specified delimiter.

e_csv

Extracts multiple fields from a specified field by using a specified delimiter and predefined field names. The default delimiter is a comma (,).

e_tsv

Extracts multiple fields from a specified field by using a specified delimiter and predefined field names. The default delimiter is \t.

e_psv

Extracts multiple fields from a specified source field by using a specified delimiter and predefined field names. The default delimiter is a vertical bar (|).

e_syslogrfc

Calculates the values of the facility and severity fields and returns the value of the facilitylabel field that indicates level information. The function calculates the values based on the value of the priority field and the specified syslog protocol.

e_anchor

Extracts strings by using the rules specified by anchor_rules.

Mapping and enrichment functions

e_dict_map

Maps the value of an input field to a value in a specified dictionary and returns a new field.

e_table_map

Maps the value of an input field to a row in a specified table and returns a new field.

e_tablestore_map

Enriches a raw log by using a data table in Tablestore as the dimension table.

e_redis_map

Enriches a raw log by using a data table in ApsaraDB for Redis as the dimension table.

e_search_dict_map

Searches the keywords in a specified dictionary for a raw log field, maps the field to a value in the dictionary, and returns a new field. The keywords must be query strings.

e_search_table_map

Searches a specified column in a specified table for a raw log field, maps the field to a row in the table, and returns a new field. The values of the column must be query strings.

Value-added content functions

e_threat_intelligence

Obtains the threat intelligence for an IP address or a domain name that is specified by a log field and assigns the threat intelligence as a value to a specified field.

Expression functions

Category

Function

Description

Event check functions

e_has

Checks whether a field exists.

e_not_has

Checks whether a field does not exist.

e_search

Searches for a log by using a query syntax that is similar to Lucene.

e_match, e_match_all, and e_match_any

Checks whether the value of a log field meets the conditions specified in a regular expression.

Operator functions

op_if

Returns the value of an expression based on a specified condition.

op_ifnull

Returns the value of the first expression whose value is not None.

op_coalesce

Returns the value of the first expression whose value is not None.

op_nullif

Returns none if the value of Expression 1 is equal to the value of Expression 2. If the values of Expression 1 and Expression 2 are different, the value of Expression 1 is returned.

op_and

Evaluates the specified expressions by using the logical AND operator and returns True if all specified expressions evaluate to true. The value of each expression can be of an arbitrary data type.

op_not

Evaluates a specified expression by using the logical NOT operator and returns the inverse Boolean value of the specified expression. The value of the expression can be of an arbitrary data type.

op_or

Evaluates the specified expressions by using the logical OR operator, and returns True if a specified expression evaluates to true or returns False if all specified expressions evaluate to false. The value of each expression can be of an arbitrary data type.

op_eq

Returns True or False based on the a==b condition.

op_ge

Returns True or False based on the a>=b condition.

op_gt

Returns True or False based on the a>b condition.

op_le

Returns True or False based on the a<=b condition.

op_lt

Returns True or False based on the a<b condition.

op_ne

Returns True or False based on the a!=b condition.

op_len

Calculates the number of characters in a text string. This function applies to strings or expressions that return tuples, lists, or dictionaries.

op_in

Checks whether a string, tuple, list, or dictionary contains a specified element and returns True or False.

op_not_in

Checks whether a string, tuple, list, or dictionary does not contain a specified element and returns True or False.

op_slice

Extracts strings from a specified string, array, or tuple.

op_index

Returns the element that corresponds to the index of a specified string, array, or tuple.

op_add

Calculates the sum of multiple values. The values can be strings or numbers.

op_max

Returns the largest value among the values of multiple fields or expressions.

op_min

Returns the smallest value among the values of multiple fields or expressions.

Conversion functions

ct_int

Converts the value of a field or an expression to an integer.

ct_float

Converts the value of a field or an expression to a floating-point number.

ct_str

Converts the value of a field or an expression to a string.

ct_bool

Converts the value of a field or an expression to a Boolean value.

ct_chr

Converts the ANSI or Unicode value of a field or an expression to a character.

ct_ord

Converts the value of a field or an expression to an ANSI value or a Unicode value.

ct_hex

Converts the value of a field or an expression to a hexadecimal number.

ct_oct

Converts the value of a field or an expression to an octal number.

ct_bin

Converts the value of a field or an expression to a binary number.

bin2oct

Converts a binary byte string to an octal string.

bin2hex

Converts a binary byte string to a hexadecimal string.

Arithmetic functions

op_abs

Returns the absolute value of an input value.

op_div_floor

Returns the integer part of the quotient of two input values.

op_div_true

Returns the quotient of two input values.

op_pow

Returns an input value raised to a specified power.

op_mul

Returns the product of two input values.

op_neg

Returns the opposite number of an input value.

op_mod

Returns the remainder of an input value divided by the other input value.

op_sub

Returns the difference between two input values.

op_round

Returns an input value rounded.

op_sum

Returns the sum of input values.

mat_ceil

Rounds an input value rounded up to the nearest integer.

mat_exp

Returns Euler's number raised to the power of an input value.

mat_fabs

Returns the absolute value of an input value.

mat_floor

Rounds an input value down to the nearest integer.

mat_log

Returns the logarithm of an input value with the base specified by the other input value.

mat_log10

Returns the base-10 logarithm of an input value.

mat_sqrt

Returns the square root of an input value.

mat_degrees

Converts radians to degrees.

mat_radians

Converts degrees to radians.

mat_sin

Returns the sine of an input value in radians.

mat_cos

Returns the cosine of an input value in radians.

mat_tan

Returns the tangent of an input value in radians.

mat_acos

Returns the arc cosine of an input value in radians.

mat_asin

Returns the arc sine of an input value in radians.

mat_atan

Returns the arc tangent of an input value in radians.

mat_atan2

Returns the arc tangent of X and Y coordinates.

mat_atanh

Returns the inverse hyperbolic tangent of an input value.

mat_hypot

Returns the Euclidean norm of two input values.

String functions

str_format

Formats strings.

str_join

Concatenates input strings to generate a new string by using a specified connector.

str_zip

Concurrently splits two values or strings that are returned by expressions and combines the results into one string.

str_encode

Encodes a string by using a specified encoding format.

str_decode

Decodes an input value by using a specified encoding format.

str_hex_escape_encode

Escapes special characters. The function can escape hexadecimal characters to Chinese characters.

str_sort

Sorts a specified object.

str_reverse

Reverses a string.

str_replace

Replaces an existing string with a specified string based on a specified rule.

str_logtash_config_normalize

Converts data in the Logstash configuration language to the JSON format.

str_translate

Replaces specified characters in a string with mapping characters.

str_strip

Deletes specified characters from a string.

str_lstrip

Deletes specified characters from the start of a string.

str_rstrip

Deletes specified characters from the end of a string.

str_lower

Converts all uppercase letters in a string to lowercase letters.

str_upper

Converts all lowercase letters in a string to uppercase letters.

str_title

Capitalizes the first letter of each word in a string and converts the other letters in the string to lowercase letters.

str_capitalize

Capitalizes the first letter of a string and converts the other letters in the string to lowercase letters.

str_swapcase

Converts the uppercase letters in a string to lowercase letters and lowercase letters to uppercase letters.

str_count

Counts the number of occurrences of a character in a string.

str_find

Checks whether a string contains a specified substring.

str_rfind

Returns the position of the last occurrence of a specified character in a string.

str_endswith

Checks whether a string ends with a specified suffix.

str_startswith

Checks whether a string starts with a specified string.

str_split

Splits a string by using a specified delimiter.

str_splitlines

Splits a string by using a line feed.

str_partition

Splits a string into three parts from left to right by using a specified delimiter.

str_rpartition

Splits a string into three parts from right to left by using a specified delimiter.

str_center

Pads a string to a specified length by using a specified character.

str_ljust

Pads a string to a specified length by using a specified character from the end of the string.

str_rjust

Pads a string to a specified length by using a specified character from the start of the string.

str_zfill

Pads a string to a specified length by using 0 from the start of the string.

str_expandtabs

Converts \t in a string to spaces.

str_isalnum

Checks whether a string contains only letters and digits.

str_isalpha

Checks whether a string contains only letters.

str_isascii

Checks whether a string is in the ASCII table.

str_isdecimal

Checks whether a string contains only decimal characters.

str_isdigit

Checks whether a string contains only digits.

str_isidentifier

Checks whether a string is a valid Python identifier or checks whether a variable name is valid.

str_islower

Checks whether a string contains lowercase letters.

str_isnumeric

Checks whether a string contains digits.

str_isprintable

Checks whether all characters in a string are printable characters.

str_isspace

Checks whether a string contains only spaces.

str_istitle

Checks whether the first letter of each word in a string is in uppercase and the other letters in the string are in lowercase.

str_isupper

Checks whether all letters in a string are in uppercase.

str_uuid

Generates a random universally unique identifier (UUID).

Date and time functions

dt_parse

Converts a value or the value of a time expression to a datetime object.

dt_str

Converts a value or the value of a time expression to a string.

dt_parsetimestamp

Converts a value or the value of a time expression to a UNIX timestamp.

dt_prop

Returns a specific attribute of a value, or returns a specific attribute of the value of a time expression. The attribute can be day or year.

dt_now

Returns the current date and time.

dt_today

Return only the current date.

dt_utcnow

Returns the current datetime object in the current time zone.

dt_fromtimestamp

Converts a UNIX timestamp to a datetime object.

dt_utcfromtimestamp

Converts a UNIX timestamp to a datetime object in the current time zone.

dt_strptime

Parses a time string into a datetime object.

dt_currentstamp

Returns the current UNIX timestamp.

dt_totimestamp

Converts a datetime object to a UNIX timestamp.

dt_strftime

Converts a datetime object to a string in a specified format.

dt_strftimestamp

Converts a UNIX timestamp to a string in a specified format.

dt_truncate

Extracts a time value from a value or the value of a time expression based on a specified time granularity.

dt_add

Changes a value or the value of a time expression based on a specified time granularity.

dt_MO

Offsets a specified time to the date of the previous or following Nth Monday. The offset value N is passed to the weekday parameter of the dt_add function.

dt_TU

Offsets a specified time to the date of the previous or following Nth Tuesday. The offset value N is passed to the weekday parameter of the dt_add function.

dt_WE

Offsets a specified time to the date of the previous or following Nth Wednesday. The offset value N is passed to the weekday parameter of the dt_add function.

dt_TH

Offsets a specified time to the date of the previous or following Nth Thursday. The offset value N is passed to the weekday parameter of the dt_add function.

dt_FR

Offsets a specified time to the date of the previous or following Nth Friday. The offset value N is passed to the weekday parameter of the dt_add function.

dt_SA

Offsets a specified time to the date of the previous or following Nth Saturday. The offset value N is passed to the weekday parameter of the dt_add function.

dt_SU

Offsets a specified time to the date of the previous or following Nth Sunday. The offset value N is passed to the weekday parameter of the dt_add function.

dt_astimezone

Converts a value or the value of a time expression to a datetime object in a specified time zone.

dt_diff

Returns the difference between two values or between the values of two time expressions based on a specified time granularity.

Regular expression functions

regex_select

Extracts a value that matches a regular expression.

regex_findall

Extracts all values that match a regular expression.

regex_match

Checks whether a value matches a regular expression.

regex_replace

Replaces the characters that match a regular expression in a string.

regex_split

Splits a string into an array of strings.

Grok function

grok

Extracts a value that matches a regular expression.

Structured data functions

json_select

Extracts or calculates specific values from a JSON expression by using JMESPath.

json_parse

Parses a value into a JSON object.

xml_to_json

Converts XML data to JSON data, and then expands the converted data.

IP address parsing functions

geo_parse

Identifies the city, province, and country based on an IP address.

ip_cidrmatch

Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.

ip_version

Checks whether the version of an IP address is IPv4 or IPv6.

ip_type

Identifies the type of an IP address and checks whether the type of the IP address is private or public.

ip_makenet

Converts an IP address to a CIDR block.

ip_to_format

Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.

ip_overlaps

Checks whether two CIDR blocks overlap.

ip2long

Converts an IP address to a value of the long type.

long2ip

Converts a value of the long type to an IP address.

Encoding and decoding functions

url_encoding

Performs URL encoding on data.

url_decoding

Performs URL decoding on data.

protobuf_decoding

Parses data into the JSON format by using a specified Protobuf template.

str_encode

Encodes a string by using a specified encoding format.

str_decode

Decodes an input value by using a specified encoding format.

base64_encoding

Encodes data by using the Base64 algorithm.

base64_decoding

Decodes data by using the Base64 algorithm.

html_encoding

Encodes data in the HTML format.

html_decoding

Decodes HTML-encoded data.

md5_encoding

Encodes data by using the MD5 algorithm.

sha1_encoding

Encodes data by using the SHA1 algorithm.

crc32_encoding

Calculates a cyclic redundancy check (CRC) code for data.

gzip_compress

Compresses and encodes data.

gzip_decompress

Decompresses compressed data.

zlib_compress

Compresses and encodes data.

zlib_decompress

Decompresses compressed data.

aes_encrypt

Encrypts data by using the AES algorithm.

aes_decrypt

Decrypts data by using the AES algorithm.

jwt_encoding

Encodes JSON data based on the JSON Web Token (JWT) standard.

jwt_decoding

Decodes data to raw JSON data based on the JWT standard.

hashids_encoding

Encodes data by using the Hashids library.

hashids_decoding

Decodes data that is encoded by using the Hashids library.

Parsing functions

ua_parse_device

Parses User-Agent and returns the device information.

ua_parse_os

Parses User-Agent and returns the operating system information.

ua_parse_agent

Parses User-Agent and returns the browser information.

ua_parse_all

Parses User-Agent and returns all information.

url_parse

Parses a URL and returns the components of the URL.

url_parse_qs

Parses the query string of a URL and returns the components of the query string.

List functions

lst_make

Constructs a list.

lst_insert

Inserts elements to a specified position in a list.

lst_append

Appends elements to a list.

lst_delete_at

Deletes the element at a specified position from a list.

lst_reverse

Reverses the order of elements in a list.

lst_get

Returns the element at a specified position in a list or a tuple.

Dictionary functions

dct_make

Constructs a dictionary.

dct_update

Updates a dictionary.

dct_delete

Deletes key-value pairs from a dictionary.

dct_keys

Returns the keys of a dictionary.

dct_values

Returns the values of a dictionary.

dct_get

Returns the value that corresponds to a specified key in a dictionary.

Table functions

tab_parse_csv

Constructs a table from comma-separated values (CSV) text.

tab_to_dict

Constructs a dictionary from a table.

Resource functions

res_local

Pulls the values of advanced parameters from the current data transformation job.

res_rds_mysql

Pulls data from a specified table in a database that is created on an ApsaraDB RDS for MySQL instance or obtains the execution result of an SQL statement. The data and result can be updated at regular intervals.

res_log_logstore_pull

Pulls data from another Logstore when you transform data in a Logstore. You can pull data in a continuous manner.

res_oss_file

Pulls data from an object in a specified Object Storage Service (OSS) bucket. The data can be updated at regular intervals.