Data structures

Data transformation uses several data structures for processing log data, including basic types, event types, fixed identifiers, and JSON objects.

Basic data structures

The following table lists the basic data structure types.

Type	Description
Integer	Sets field values or passes parameters to functions. Example: `e_set("f1", 100)` sets field `f1` to 100.
Float	Sets field values or passes parameters to functions. Example: `e_set("f1", 1.5)` sets field `f1` to 1.5.
String	Strings support the following conventions: `"abc"` is the same as `'abc'`. If a string contains a double quotation mark ("), you can use single quotation marks to enclose the string, such as `'abc"xyz'`. You can also use a backslash (\) to escape the double quotation mark, such as `"abc\"xyz"`. A backslash (\) is an escape character. For example, `"\\abc\\xyz"` represents the string `\abc\xyz`. `r"\\10.64.1.1\share\folder"` is the same as `"\\\\10.64.1.1\\share\\folder"`. Both represent the string `\\10.64.1.1\share\folder`. Multi-byte strings are represented in Unicode. For example, the length of the string `Chinese` is 2. Regular expressions are represented as strings. Note Use single quotation marks for the outer string and double quotation marks for the inner search value. `e_search("domain: '/url/test.jsp'")` is incorrect. The correct format is `e_search('domain: "/url/test.jsp"')`.
Bytes	Differs from string memory encoding. `b'abc'` is used as a parameter or return type for some special functions.
empty	`None` or `null` indicates an absence of value. Some named parameters in functions default to `None`, which indicates a specific default behavior. Note An empty string has a different data type than None or null.
List	An array, such as `[1,2,3,4]`. Some functions accept lists as parameters. Example: `e_dict_map("dict data", ["f1", "f2", "f3"], ...)`. Some functions return lists. Example: `json_select` returns a list when selecting an array.
Tuple	Functions the same as a list. Example: `(1,2,3,4)`.
Dictionary (Dict)	A collection of key-value pairs in the format `{"key": "value", "k2": "v2", ...}`. Keys are typically unique strings. Values can be any type listed above. Dictionaries use hashing and are unordered. An event is a special dictionary type. Some functions accept dictionaries in specific formats. Example: `{"key": [1,2,3], "key2": {"k3": "v3"}}`. Dictionaries are also used as input data in dictionary mappings.
Boolean (Bool)	For example, `True`, `False`, `true`, or `false`.
Table	A multi-column table structure. Build tables by loading multi-row CSV content from external resources or retrieving multi-column data from ApsaraDB RDS or a Logstore. Tables are mainly used for data enrichment mappings or other advanced configuration scenarios.
Datetime object	Represents a date and time in memory. Can be converted to a Unix timestamp string or a formatted time string, or passed to `dt_` functions for further transformation.

Event types

Data transformation supports the following event types.

Primitive type

Data transformation processes log data as a dictionary, such as {"__topic__": "access_log", "content": "....."}.

Dictionary keys and values correspond to log fields and values.

Note
Event keys and values are both strings, and keys must be unique.
Meta-fields
Meta-fields include:
- Time field __time__: the Unix timestamp (seconds since 00:00:00 UTC, January 1, 1970) when the log was written. Stored as an integer string.
- Topic __topic__: organizes logs in a Logstore by topic. You can specify a topic when writing or querying logs.
- Source __source__: the log source, such as the IP address of the machine that generated the log.
Time field modification

Modify the time field value to change the event time. You can also use datetime functions for other operations on this field.

Note
If the time field is deleted, the system uses the processing time as the event time when sending the log to the destination.
Tags
A tag is a log label used for classification, in the format __tag__:name.
- If public IP address recording is enabled for the source Logstore, logs include the tag:__tag__:__receive_time__ tag.
- Container Service logs contain multiple container-related tags, such as __tag__:__container_name__.
- You can add or modify tags. To add a tag named type, use e_set("__tag__:type", "access_log").

Automatic conversion on assignment

Event keys and values are strings. When you assign a value or set a new field, keys and values are automatically converted to strings. Example:

e_set("v1", 12.3)
e_set("v2", True)

This sets v1 to string "12.3" and v2 to string "true".

The following table shows how different data types convert to strings.

Type	Example	Converted type	Converted example
Integer	`1`	String	`"1"`
Float	`1.2`	String	`"1.2"`
Boolean	`True`	String	`"true"`
Bytes	`b"123"`	Decoded to a string using UTF-8	`"123"`
Tuple	Example 1: `(1, 2, 3)` Example 2: `("a", 1)`	String of a list	Example 1: `"[1, 2, 3]"` Example 2: `"[\"a\", 1]"`
List	Example 1: `[1,2,3]` Example 2: `["a", 1]`	String	Example 1: `"[1, 2, 3]"` Example 2: `"[\"a\", 1]"`
Dictionary	`{"1":2, "3":4}`	String	`"{\"1\": 2, \"3\": 4}"`
Datetime	`datetime(2018, 10, 10, 10, 10, 10)`	ISO format string	`2018-10-10 10:10:10`

Fixed identifiers

Data transformation provides fixed identifiers to simplify code and improve readability.

Identifier	Type	Description
true	Boolean	Equivalent to `True`.
false	Boolean	Equivalent to `False`.
null	None	Equivalent to `None`.
F_TAGS	String	Regex for `TAG` fields. Equivalent to `"__tag__:.+"`.
F_META	String	Regex for `TAG`, `__topic__`, and `__source__` fields. Equivalent to `__tag__:.+\|__topic__\|__source__`.
F_TIME	String	The `__time__` field name. Equivalent to `__time__`.
F_PACK_META	String	Regex for `pack meta` fields. Equivalent to `"__pack_meta__\|__tag__:__pack_id__"`.
F_RECEIVE_TIME	String	The `tag` field for the server log receive time. Equivalent to `"__tag__:__receive_time__"`.
C_JOB_REGION	String	The region ID of the data transformation task, such as `cn-hangzhou`. Example: `e_set("job_region", C_JOB_REGION)`.
C_JOB_PROJECT	String	The project name of the data transformation task, such as `my-sls-project`. Example: `e_set("job_project", C_JOB_PROJECT)`.
C_JOB_NAME	String	The data transformation job configuration name, such as `etl-1649227848-642277`. Example: `e_set("job_name", C_JOB_NAME)`.
C_JOB_ID	String	The run ID of the data transformation task, such as `73b96061b8c1c2101d558139bf641ea9`. Example: `e_set("job_id", C_JOB_ID)`.

JSON objects

A JSON object is parsed from a string using json_select or json_parse. It is composed of basic data structures. The following table shows how input strings convert to JSON objects.

String	JSON object	Actual type
`1`	`1`	Integer
`1.2`	`1.2`	Float
`true`	`True`	Boolean
`false`	`False`	Boolean
`"abc"`	`"abc"`	String
`null`	`None`	None
`["v1", "v2", "v3"]`	`["v1", "v2", "v3"]`	List
`["v1", 3, 4.0]`	`["v1", 3, 4.0]`	List
`{"v1": 100, "v2": "good"}`	`{"v1": 100, "v2": "good"}`	Dictionary
`{"v1": {"v11": 100, "v2": 200}, "v3": "good"}`	`{"v1": {"v11": 100, "v2": 200}, "v3": "good"}`	Dictionary