Dictionaries and tables are two major types of data structures used for data enrichment. This topic describes common methods for building dictionaries and tables and compares the advantages and disadvantages of different building methods.

Build dictionaries

  • Build a dictionary directly
    e_dict_map({"400": "error", "200": "ok", "*": "other"}, "status", "message")
  • Build a dictionary from data configuration items
    e_dict_map(res_local("http_code_map"), "status", "message")
    where, http_code_map is an advanced task configuration item. Its values are as follows:
    {"400": "error", "200": "ok", "*": "other"}
  • Build a dictionary from a table
    Use the tab_to_dict function to build a dictionary from a table. Details on how to build a table are available in later part of this topic.
    e_dict_map(tab_to_dict(tab_parse_csv("status_code,status_info\n400,error\n200,ok\n*,other"), "status_code", "status_info"), "status", "message")
  • Build a dictionary by using the dct_make function
    e_dict_map(dct_make("400", "error", "200", "ok", "*", "other"), "status", "message")
  • Build a dictionary by using another expression
    e_dict_map(json_parse(v("http_code_map")), "status", "message")

    where, the data mappings are obtained from the http_code_map field in source logs.

The following table compares these dictionary building methods.
Method Advantage Disadvantage
Build a dictionary directly This method is intuitive, simple, and easy to use. The rules will be lengthy if too much content is involved. In addition, the built dictionary is static and inflexible.
Build a dictionary from data configuration items We recommend that you use this method if the dictionary contains a large amount of data and is frequently modified. This method features easy maintenance. A dictionary built in this method is not scalable or reusable across tasks. In addition, it does not support automatic update.
Build a dictionary from a table Featuring flexible maintenance, this method is used in advanced scenarios. You need to build and maintain a table for building a dictionary, making the configuration process complicated.
Build a dictionary by using the dct_make function You can build a dictionary based on dynamic logic in specific scenarios. This method is relatively advanced and difficult to maintain.
Build a dictionary by using another expression This method dynamically extracts data mappings from fields in log events. It applies in specific scenarios. This method is relatively advanced and difficult to maintain.

Build tables

  • Build a table from a text file
    e_table_map(tab_parse_csv("city,name,age\nshanghai,aliyun,10\ncity:nanjing,Maki,18"), "name",["city", "age"])
  • Build a table from data configuration items
    e_search_table_map(tab_parse_csv(res_local("table_info")), "name",["city", "age"])
    where, table_info is a task configuration item in data transformation rules. Its values are as follows:
    content,name,age
    shanghai,aliyun,10
    nanjing,Maki,18
  • Build a table from an ApsaraDB for Relational Database Service (RDS) database
    e_table_map(tab_parse_csv(res_rds_mysql(...database="db", table="city")), "name",["city", "age"])
    The city table in the RDS database contains the following information:
    content,name,age
    shanghai,aliyun,10
    nanjing,Maki,18
  • Build a table from another Logstore
    e_table_map(res_log_logstore_pull(..., project="project_name", logstore="logstore_name", fields=["city","name","age"]),, "name",["city", "age"])
    The specified Logstore contains the following log events:
    "Log 1"
    {
      "city": "shanghai",
      "name": "aliyun",
      "age": "10"
    }
    "Log 2"
    {
      "city": "city:nanjing and data > 100",
      "name": "Maki",
      "age": "18"
    }
The following table compares these table building methods.
Method Advantage Disadvantage
Build a table from a text file This method is intuitive, simple, and easy to use. The rules will be lengthy if too much content is involved. A table built in this method is difficult to maintain, scale up, or reuse.
Build a table from data configuration items We recommend that you use this method if the table contains a large amount of data and is frequently modified. This method features easy maintenance. A table built in this method is not scalable or reusable across tasks. In addition, it does not support automatic update.
Build a table from an ApsaraDB for RDS database
  • We recommend that you use this method if the table contains a large amount of data and is frequently modified. This method features easy maintenance.
  • This method supports automatic update.
  • A table built in this method can be reused across tasks.
You need to connect to an external ApsaraDB for RDS database for building a table, making the configuration process complicated.
Build a table from another Logstore This method reads data in real time. Featuring flexible maintenance, this method is used in advanced scenarios. You need to connect to another Logstore for building a table, making the configuration process complicated.