All Products
Search
Document Center

Parse JSON logs

Last Updated: Mar 04, 2022

If the stdout logs of containers in elastic container instances are Kubernetes logs, and Kubernetes automatically includes the timestamp and source of the logs in each log entry, the logs may fail to be parsed. For example, if Kubernetes automatically includes prefixes in JSON stdout logs, the logs fail to be parsed. This topic describes how to use the processor capability of Alibaba Cloud Log Service to parse JSON logs.

Problem description

The stdout logs that are generated by using the base components of elastic container instances use the same format as Kubernetes logs. Kubernetes automatically includes the timestamp and source of the logs in each entry of stdout logs. This may change the format of Kubernetes logs and cause the system to fail to parse JSON logs. Example:

2020-04-02T15:40:05.440500764+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:07.442412564+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:09.442774495+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:11.443799303+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:13.445099622+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:15.445934358+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:17.447064707+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:19.448112987+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:21.449393263+08:00 stdout F {"key1":"val1","key2":"val2"}

Solutions

You can use the processor capability of Log Service to parse JSON logs. Perform the following operations:

  1. Log on to the Log Service console.

  2. In the Projects section, click the name of the project in which the Logstore resides.

  3. On the Logstores tab, find the Logstore in which the logs are stored, click the Hide/Show icon icon to the left of the Logstore name, and then click the icon to the left of Logtail Configurations.

  4. Click the Logtail configuration that you want to modify, and then click Modify in the upper-right corner of the page.

  5. On the page that appears, set processors parameters in the Plug-in Config section and click Save.

    Select Simple Mode for Mode, turn on the Enable Plug-in Processing switch, and then set the processors parameters in the Plug-in Config section.

    Parse JSON logs

    The following sample code shows how to configure processors:

    {
        "processors": [
            {
                "type": "processor_anchor",
                "detail": {
                    "SourceKey": "content",
                    "Anchors": [
                        {
                            "Start": "stdout F ",
                            "Stop": "",
                            "FieldName": "json_content",
                            "FieldType": "string",
                            "ExpondJson": false
                        }
                    ]
                }
            },
            {
                "type": "processor_json",
                "detail": {
                    "SourceKey": "json_content",
                    "KeepSource": false,
                    "ExpandConnector": ""
                }
            }
        ]
    }

    Save the Logtail configurations and wait until the system displays the logs that are parsed as expected.

    JSON