This topic describes how to enrich log data by using mapping functions such as e_dict_map and e_search_dict_map.
Background information
- Common mapping functions map data by using the full-text matching method. Common mapping
functions include the e_dict_map and e_table_map functions. The input data of the
e_dict_map function is in the dictionary format. The input data of the e_table_map
function is in the format of a table obtained by using resource functions. For more
information about the e_dict_map function, see e_dict_map
%;. For more information about the e_table_map function, see e_table_map. For more information about resource functions, see Resource functions.For example, you can use the e_dict_map function to transform HTTP status codes in nginx logs into data of the Text type.HTTP status code Text 200 Success 300 Redirect 400 Request error 500 Server error - Search mapping functions use query strings to map fields. You can specify regular expressions or wildcard characters in query
strings and use the exact match or fuzzy match method to map data. Search mapping
functions include the e_search_dict_map and e_search_table_map functions. The input
data of the e_search_dict_map function is in the dictionary format. The input data
of the e_search_table_map function is in the format of a table obtained by using resource
functions. For more information about the e_dict_map function, see e_search_dict_map. For more information about the e_table_map function, see e_search_table_map. For more information about resource functions, see Resource functions.
For example, you can use the e_search_dict_map function to transform HTTP status codes that match the specified patterns in nginx logs into data of the Text type.
HTTP status code Text 2XX Success 3XX Redirect 4XX Request error 5XX Server error
Use the e_dict_map function to enrich log data
This section describes how to use the e_dict_map function to enrich log data.
- Raw log entry
http_host: example.com http_status: 300 request_method: GET http_host: example.org http_status: 200 request_method: POST http_host: example.net http_status: 400 request_method: GET http_host: aliyundoc.com http_status: 500 request_method: GET - Transformation requirements
Transform the status codes in the http_status field into data of the Text type and add the transformed data to the status_desc field.
- Transformation rule
e_dict_map({"400": "Request error", "500": "Server error", "300": "Redirect", "200": "Success"}, "status", "status_desc")Note The preceding transformation rule includes only four HTTP status codes. For more information, see HTTP status codes. If the value of the http_status field is 401 or 404, the corresponding value must be included in the source dictionary. Otherwise, the data mapping will fail. - Result
http_host: example.com http_status: 300 request_method: GET status_desc: Redirect http_host: example.org http_status: 200 request_method: POST status_desc: Success http_host: example.net http_status: 400 request_method: GET status_desc: Request error http_host: aliyundoc.com http_status: 500 request_method: GET status_desc: Server error
Use the e_search_dict_map function to enrich log data
This section describes how to use the e_search_dict_map function to enrich log data.
- Raw log entry
http_host: example.com http_status: 200 request_method: GET body_bytes_sent: 740 http_host: example.org http_status: 200 request_method: POST body_bytes_sent: 1123 http_host: example.net http_status: 404 request_method: GET body_bytes_sent: 711 http_host: aliyundoc.com http_status: 504 request_method: GET body_bytes_sent: 1822 - Transformation requirements
Add a field named type to each log entry. The value of this field is decided based on the values of the http_status and body_bytes_sent fields in each log entry.
- If the value of the http_status field matches the 2XX pattern and the value of the body_bytes_sent field is less than 1000 in a log entry, set the value of the type added to the log entry to Normal.
- If the value of the http_status field matches the 2XX pattern and the value of the body_bytes_sent field is equal to or greater than 1000 in a log entry, set the value of the type field added to the log entry to Too long.
- If the value of the http_status field in a log entry matches the 3XX pattern, set the value of the type field added to the log entry to Redirect.
- If the value of the http_status field in a log entry matches the 4XX pattern, set the value of the type field added to the log entry to Error.
- If the value of the http_status field in a log entry does not match either of the preceding patterns, set the value of the type field added to the log entry to Others.
- Transformation rule
e_search_dict_map({'http_status~="2\d+" and body_bytes_sent < 1000': "Normal", 'http_status~="2\d+" and body_bytes_sent >= 1000': "Too long", 'http_status~="3\d+"': "Redirect", 'http_status~="4\d+"': "Error", "*": "Others"}, "http_status", "type")If you want to use a dictionary to enrich your log data, you can create a dictionary by using braces ({}) or based on resources allocated to the task, Object Storage Service (OSS) resources, and tables. For more information, see Build dictionaries.
- Result
type: Normal http_host: example.com http_status: 200 request_method: GET body_bytes_sent: 740 type: Too long http_host: example.org http_status: 200 request_method: POST body_bytes_sent: 1123 type: Error http_host: example.net http_status: 404 request_method: GET body_bytes_sent: 711 type: Others http_host: aliyundoc.com http_status: 504 request_method: GET body_bytes_sent: 1822