All Products
Search
Document Center

Simple Log Service:Build dictionaries and tables for data enrichment

Last Updated:Dec 29, 2024

Dictionaries and tables are two major types of data structures used for data enrichment. This topic describes common methods for building dictionaries and tables and compares the advantages and disadvantages of different building methods.

Build dictionaries

  • Build a dictionary directly

    e_dict_map({"400": "error", "200": "ok", "*": "other"}, "status", "message")
  • Build a dictionary from data configuration items

    e_dict_map(res_local("http_code_map"), "status", "message")

    where, http_code_map is an advanced task configuration item. Its values are as follows:

    {"400": "error", "200": "ok", "*": "other"}
  • Build a dictionary from a table

    Use the tab_to_dict function to build a dictionary from a table. Details on how to build a table are available in later part of this topic.

    e_dict_map(tab_to_dict(tab_parse_csv("status_code,status_info\n400,error\n200,ok\n*,other"), "status_code", "status_info"), "status", "message")
  • Build a dictionary by using the dct_make function

    e_dict_map(dct_make("400", "error", "200", "ok", "*", "other"), "status", "message")
  • Build a dictionary by using another expression

    e_dict_map(json_parse(v("http_code_map")), "status", "message")

    where, the data mappings are obtained from the http_code_map field in source logs.

The following table compares these dictionary building methods.

Method

Advantage

Disadvantage

Build a dictionary directly

This method is intuitive, simple, and easy to use.

The rules will be lengthy if too much content is involved. In addition, the built dictionary is static and inflexible.

Build a dictionary from data configuration items

We recommend that you use this method if the dictionary contains a large amount of data and is frequently modified. This method features easy maintenance.

A dictionary built in this method is not scalable or reusable across tasks. In addition, it does not support automatic update.

Build a dictionary from a table

Featuring flexible maintenance, this method is used in advanced scenarios.

You need to build and maintain a table for building a dictionary, making the configuration process complicated.

Build a dictionary by using the dct_make function

You can build a dictionary based on dynamic logic in specific scenarios.

This method is relatively advanced and difficult to maintain.

Build a dictionary by using another expression

This method dynamically extracts data mappings from fields in log events. It applies in specific scenarios.

This method is relatively advanced and difficult to maintain.

Build tables

  • Build a table from a text file

    e_table_map(tab_parse_csv("city,name,age\nshanghai,aliyun,10\ncity:nanjing,Maki,18"), "name",["city", "age"])
  • Build a table from an ApsaraDB RDS database

    e_table_map(tab_parse_csv(res_rds_mysql(...database="db", table="city")), "name",["city", "age"])

    The city table in the RDS database contains the following information:

    content,name,age
    shanghai,aliyun,10
    nanjing,Maki,18
  • Build a table from another logstore

    e_table_map(res_log_logstore_pull(..., project="project_name", logstore="logstore_name", fields=["city","name","age"]),, "name",["city", "age"])

    The specified logstore contains the following log events:

    "Log 1"
    {
      "city": "shanghai",
      "name": "aliyun",
      "age": "10"
    }
    "Log 2"
    {
      "city": "city:nanjing and data > 100",
      "name": "Maki",
      "age": "18"
    }

The following table compares these table building methods.

Method

Advantage

Disadvantage

Build a table from a text file

This method is intuitive, simple, and easy to use.

The rules will be lengthy if too much content is involved. A table built in this method is difficult to maintain, scale up, or reuse.

Build a table from an ApsaraDB RDS database

  • We recommend that you use this method if the table contains a large amount of data and is frequently modified. This method features easy maintenance.

  • This method supports automatic update.

  • A table built in this method can be reused across tasks.

You need to connect to an external ApsaraDB RDS database for building a table, making the configuration process complicated.

Build a table from another logstore

This method reads data in real time. Featuring flexible maintenance, this method is used in advanced scenarios.

You need to connect to another logstore for building a table, making the configuration process complicated.