All Products
Search
Document Center

Elastic Container Instance:Parse JSON logs

Last Updated:Mar 22, 2024

stdout logs of containers in elastic container instances are native Kubernetes logs. Kubernetes automatically prefixes the timestamp and source of a log to each log entry. As a result, the system cannot parse JSON logs. This topic describes how to use the processor capability of Simple Log Service to parse the JSON logs of Elastic Container Instance.

Problem description

The stdout logs that are generated by using the base components of elastic container instances use the same format as Kubernetes logs. Kubernetes automatically prefixes the timestamp and source of a log to each entry of stdout logs. This may change the format of Kubernetes logs and cause the parse failure of JSON logs. Example:

2020-04-02T15:40:05.440500764+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:07.442412564+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:09.442774495+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:11.443799303+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:13.445099622+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:15.445934358+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:17.447064707+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:19.448112987+08:00 stdout F {"key1":"val1","key2":"val2"}
2020-04-02T15:40:21.449393263+08:00 stdout F {"key1":"val1","key2":"val2"}

Solution

You can use the processor capability of Simple Log Service to parse JSON logs. Procedure:

  1. Log on to the Simple Log Service console.

  2. In the Projects section, click the name of the project in which the Logstore resides.

  3. On the Logstores tab, find the Logstore in which the logs are stored, click the 展开图标 icon to the left of the Logstore name, and then click Logtail Configurations.

  4. Click the Logtail configuration that you want to modify, and then click Modify in the upper-right corner of the page.

  5. On the page that appears, configure processor parameters in the Plug-in Config section and click Save.

    Select Simple Mode for Mode, turn on Enable Plug-in Processing, and add processor-related configurations in the Plug-in Config section.

    JSON日志解析

    The following sample code shows how to configure processors:

    {
        "processors": [
            {
                "type": "processor_anchor",
                "detail": {
                    "SourceKey": "content",
                    "Anchors": [
                        {
                            "Start": "stdout F ",
                            "Stop": "",
                            "FieldName": "json_content",
                            "FieldType": "string",
                            "ExpondJson": false
                        }
                    ]
                }
            },
            {
                "type": "processor_json",
                "detail": {
                    "SourceKey": "json_content",
                    "KeepSource": false,
                    "ExpandConnector": ""
                }
            }
        ]
    }

    Save the Logtail configurations and wait until the parsed logs are displayed.

    JSON