Dictionaries and tables are two major types of data structures used for data enrichment. This topic describes common methods for building dictionaries and tables and compares the advantages and disadvantages of different building methods.
Build dictionaries
Build a dictionary directly
e_dict_map({"400": "error", "200": "ok", "*": "other"}, "status", "message")
Build a dictionary from data configuration items
e_dict_map(res_local("http_code_map"), "status", "message")
where,
http_code_map
is an advanced task configuration item. Its values are as follows:{"400": "error", "200": "ok", "*": "other"}
Build a dictionary from a table
Use the
tab_to_dict
function to build a dictionary from a table. Details on how to build a table are available in later part of this topic.e_dict_map(tab_to_dict(tab_parse_csv("status_code,status_info\n400,error\n200,ok\n*,other"), "status_code", "status_info"), "status", "message")
Build a dictionary by using the dct_make function
e_dict_map(dct_make("400", "error", "200", "ok", "*", "other"), "status", "message")
Build a dictionary by using another expression
e_dict_map(json_parse(v("http_code_map")), "status", "message")
where, the data mappings are obtained from the
http_code_map
field in source logs.
The following table compares these dictionary building methods.
Method | Advantage | Disadvantage |
Build a dictionary directly | This method is intuitive, simple, and easy to use. | The rules will be lengthy if too much content is involved. In addition, the built dictionary is static and inflexible. |
Build a dictionary from data configuration items | We recommend that you use this method if the dictionary contains a large amount of data and is frequently modified. This method features easy maintenance. | A dictionary built in this method is not scalable or reusable across tasks. In addition, it does not support automatic update. |
Build a dictionary from a table | Featuring flexible maintenance, this method is used in advanced scenarios. | You need to build and maintain a table for building a dictionary, making the configuration process complicated. |
Build a dictionary by using the dct_make function | You can build a dictionary based on dynamic logic in specific scenarios. | This method is relatively advanced and difficult to maintain. |
Build a dictionary by using another expression | This method dynamically extracts data mappings from fields in log events. It applies in specific scenarios. | This method is relatively advanced and difficult to maintain. |
Build tables
Build a table from a text file
e_table_map(tab_parse_csv("city,name,age\nshanghai,aliyun,10\ncity:nanjing,Maki,18"), "name",["city", "age"])
Build a table from an ApsaraDB RDS database
e_table_map(tab_parse_csv(res_rds_mysql(...database="db", table="city")), "name",["city", "age"])
The
city
table in the RDS database contains the following information:content,name,age shanghai,aliyun,10 nanjing,Maki,18
Build a table from another logstore
e_table_map(res_log_logstore_pull(..., project="project_name", logstore="logstore_name", fields=["city","name","age"]),, "name",["city", "age"])
The specified logstore contains the following log events:
"Log 1" { "city": "shanghai", "name": "aliyun", "age": "10" } "Log 2" { "city": "city:nanjing and data > 100", "name": "Maki", "age": "18" }
The following table compares these table building methods.
Method | Advantage | Disadvantage |
Build a table from a text file | This method is intuitive, simple, and easy to use. | The rules will be lengthy if too much content is involved. A table built in this method is difficult to maintain, scale up, or reuse. |
Build a table from an ApsaraDB RDS database |
| You need to connect to an external ApsaraDB RDS database for building a table, making the configuration process complicated. |
Build a table from another logstore | This method reads data in real time. Featuring flexible maintenance, this method is used in advanced scenarios. | You need to connect to another logstore for building a table, making the configuration process complicated. |