All Products
Search
Document Center

Parse JSON logs

Last Updated: May 19, 2022

If the standard output logs of containers in elastic container instances are native Kubernetes logs, and Kubernetes automatically includes prefixes such as the timestamp and source of the logs in each log entry, configuration logs may fail to be parsed. For example, if Kubernetes automatically includes prefixes in JSON standard output logs, the logs fail to be parsed. This topic describes how to use the processor capability of Alibaba Cloud Log Service to parse JSON logs.

Problem description

The standard output logs that are generated by using the base components of elastic container instances use the same format as native Kubernetes logs. Kubernetes automatically includes prefixes such as the timestamp and source of the logs in each entry of standard output logs. This may change the format of the native logs and cause the system to fail to parse JSON logs. Example:

2020-04-02T15:40:05.440500764+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:07.442412564+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:09.442774495+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:11.443799303+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:13.445099622+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:15.445934358+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:17.447064707+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:19.448112987+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:21.449393263+08:00 stdout F {"key1":"val1","key2":"val2"}

Solution

You can use the processor capability of Log Service to parse JSON logs. Perform the following steps:

  1. Log on to the Log Service console.

  2. In the Projects section, click the name of the project in which the Logstore resides.

  3. On the Logstores tab, find the Logstore in which the logs are stored, click the Hide/Show icon icon to the left of the Logstore name, and then click the Hide/Show icon icon to the left of Logtail Configurations.

  4. Click the Logtail configuration that you want to modify, and then click Modify in the upper-right corner of the page.

  5. On the page that appears, specify the names of processors configurations in the Plug-in Config field and click Save.

    Select Simple Mode for Mode and turn on the Enable Plug-in Processing switch. Specify the names of the processors configurations in the Plug-in Config field.

    Parse JSON logs

    The following sample code shows how to configure processors:

    {
        "processors": [
            {
                "type": "processor_anchor",
                "detail": {
                    "SourceKey": "content",
                    "Anchors": [
                        {
                            "Start": "stdout F ",
                            "Stop": "",
                            "FieldName": "json_content",
                            "FieldType": "string",
                            "ExpondJson": false
                        }
                    ]
                }
            },
            {
                "type": "processor_json",
                "detail": {
                    "SourceKey": "json_content",
                    "KeepSource": false,
                    "ExpandConnector": ""
                }
            }
        ]
    }

    Save the Logtail configurations. Wait for a few minutes. After the waiting period elapses, the system displays the logs that are parsed as expected.

    JSON