This topic describes the data structures in data transformation syntax.
Basic data structures
The following table describes the basic data structures.
Type | Description |
---|---|
Integer | You can use integers as field values. You can also use integers to pass the values
of function parameters.
For example, the |
Float | You can use floats as field values. You can also use floats to pass the values of
function parameters.
For example, the |
String | Strings can be used in multiple formats. For example:
Note The strings used in the e_search function must be enclosed by double quotation marks
("") instead of single quotation marks (''). For example,
e_search("domain: '/url/test.jsp'") is invalid whereas e_search('domain: "/url/test.jsp"') is valid.
|
Bytes | Example: b'abc' . Bytes are encoded in memory by using a format that is different from the format
of strings. Bytes are received and returned by special functions.
|
None | Both None and null indicate a NULL value. Some named parameters of functions use None as the default value to indicate a specific default behavior.
Note None or null does not indicate an empty string.
|
List | An array, for example, [1, 2, 3, 4].
|
Tuple | Tuples are similar to lists, for example, (1,2,3,4). |
Dictionary | A collection of key-value pairs in the format of {"key": "value", "k2": "v2", ...} . A key is usually a unique string and a value can be of any data type. The key-value
pairs are stored in a hash table in an unordered manner.
|
Boolean | For example, True , False , true , and false .
|
Table | Each table consists of multiple rows and columns. You can load data in the comma-separated values (CSV) file that consists of multiple rows from an external resource to construct a table. You can also load multiple columns of data from an ApsaraDB RDS database or a Logstore to construct a table. Tables are used in advanced operations such as data mapping and enrichment. |
Datetime object | A memory object that indicates date and time. A datetime object can be converted to
a UNIX timestamp or a formatted time string. A datetime object can be passed to dt_ functions for further conversion.
|
Event structure and fields
- Basic structure
An event is processed in the dictionary structure during the data transformation process, for example,
{"__topic__": "access_log", "content": "....."}
.The keys and values of the dictionary correspond to the fields and values in the log.
Note The keys and values of an event are strings, and the keys must be unique. - Meta-fields
The following meta-fields are supported:
__time__
: the timestamp when the log is received by Log Service. The value is an integer string. The timestamp follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 Thursday, January 1, 1970.__topic__
: the topic of the log. Topics are used to group logs in a Logstore. You can specify a topic for logs that are written to Log Service. You can also specify a topic when you query logs.__source__
: the source of the log. For example, the value of this field can be the IP address of the server that is used to generate the log.
- Time field modification
You can change the value of the __time__ field to modify the event time of a log. You can use date and time functions to perform more operations on the __time__ field.Note If the __time__ field is deleted, the system time when the log is processed is used as the event time when the log is written to the destination Logstore.
- Tags
Tags are used to differentiate fields in logs. Tags are in the format of
__tag__:name
.- If the source Logstore enables the feature of recording the public IP address, logs
contain the
__tag__:__receive_time__
tag. - Container logs contain many container-related tags, such as
__tag__:__container_name__
. - You can add and modify tags. For example, you can add a tag named type by calling the
e_set("__tag__:type", "access_log")
function.
- If the source Logstore enables the feature of recording the public IP address, logs
contain the
- Automatic type conversion during assignment
The keys and values of an event are represented as strings. Therefore, when you assign a value to a field, the value is automatically converted to a string. For example:
After you call the preceding functions, thee_set("v1", 12.3) e_set("v2", True)
v1
field is set to the string 12.3, and thev2
field is set to the string true.The following table provides conversion examples of different data types.Type Value Conversion type Conversion value Integer 1
String "1"
Float 1.2
String "1.2"
Boolean True
String "true"
Byte b"123"
String encoded in UTF-8 "123"
Tuple - Example 1:
(1, 2, 3)
- Example 2:
("a", 1)
String that represents a list - Example 1:
"[1, 2, 3]"
- Example 2:
"[\"a\", 1]"
List - Example 1:
[1,2,3]
- Example 2:
["a", 1]
String - Example 1:
"[1, 2, 3]"
- Example 2:
"[\"a\", 1]"
Dictionary {"1":2, "3":4}
String "{\"1\": 2, \"3\": 4}"
Datetime datetime(2018, 10, 10, 10, 10, 10)
String that represents time in the ISO format 2018-10-10 10:10:10
- Example 1:
Identifiers
Identifier | Type | Equivalent |
---|---|---|
true | Bool | True |
false | Bool | False |
null | None | None |
F_TAGS | String | Regular expression that represents tag fields: "__tag__:. +" |
F_META | String | Regular expression that represents tag , __topic__ , and __source__ fields: __tag__:.+|__topic__|__source__ |
F_TIME | String | __time__
|
F_PACK_META | String | Regular expression that represents pack meta fields: "__pack_meta__|__tag__:__pack_id__" |
F_RECEIVE_TIME | String | "__tag__:__receive_time__"
|
JSON objects
json_select
and json_parse
. The objects are in basic data structures. For more information, see Basic data structures. The following table describes how strings are converted to JSON objects.
String | JSON object | Type |
---|---|---|
1 |
1 |
Integer |
1.2 |
1.2 |
Float |
true |
True |
Boolean |
false |
False |
Boolean |
"abc" |
"abc" |
String |
null |
None |
None |
["v1", "v2", "v3"] |
["v1", "v2", "v3"] |
List |
["v1", 3, 4.0] |
["v1", 3, 4.0] |
List |
{"v1": 100, "v2": "good"} |
{"v1": 100, "v2": "good"} |
Dictionary |
{"v1": {"v11": 100, "v2": 200}, "v3": "good"} |
{"v1": {"v11": 100, "v2": 200}, "v3": "good"} |
Dictionary |