All Products
Search
Document Center

:Collect script execution logs

Last Updated:Feb 29, 2024

This topic describes how to create a Logtail configuration in the Simple Log Service console to collect script execution logs.

Prerequisites

Logtail V1.7.1 or later is installed on your Linux server. For more information, see Install Logtail on a Linux server.

Description

You can specify a user and a script in a Logtail configuration. Logtail executes the script as the user to obtain script execution logs.

  • You can specify one of the following types of scripts: bash, shell, python2, and python3.

  • You can specify a user to execute the script. This way, you can achieve user-based access control on resources that can be accessed by the script.

  • If the script execution times out, Logtail automatically stops executing the script. Logtail does not collect the script execution logs.

Procedure

  1. Log on to the Simple Log Service console.

  2. In the Import Data section, select Custom Data Plug-in.

  3. Select a project and a Logstore. Then, click Next.

  4. Create a machine group.

    • If a machine group is available, click Use Existing Machine Groups.

    • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.

      1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.

        For more information, see Install Logtail on ECS instances.

        Important

        If your server is an ECS instance that belongs to a different Alibaba Cloud account from Simple Log Service, a server in a data center, or a server from a third-party cloud service provider, you must manually install Logtail on the server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server. After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.

      2. After Logtail is installed, click Complete Installation.

      3. In the Create Machine Group step, configure the Name parameter and click Next.

        Simple Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.

  5. Select the machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.

    Important

    If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, troubleshoot the issue by following the instructions provided in How do I troubleshoot an error that is related to a Logtail machine group in a host environment?

  6. In the Specify Data Source step, configure the Config Name and Plug-in Config parameters. Then, click Next.

    • inputs is required and is used to configure the data collection settings for the Logtail configuration. You must configure inputs based on your data source.

      Important

      You can configure only one type of data source in the inputs field.

    • processors is optional and is used to configure the data processing settings for the Logtail configuration. You can specify one or more processing methods.

      If your logs cannot be parsed based only on the setting of inputs, you can configure processors in the Plug-in Config field to add plug-ins for data processing. For example, you can extract fields, extract log time, mask data, and filter logs. For more information, see Overview.

    {
        "inputs":[
            {
                "type":"input_command",
                "detail":{
                    "ScriptType":"python2",
                    "User":"test",
                    "ScriptContent":"print(\"test input_command 0\")\nprint(\"test input_command 1\")\nprint(\"test input_command 2\")\nprint(\"test input_command 3\")\nprint(\"test input_command 4\")\nprint(\"test input_command 5\")\nprint(\"test input_command 6\")\nprint(\"test input_command 7\")\nprint(\"test input_command 8\")\nprint(\"test input_command 9\")\n",
                    "ContentEncoding":"PlainText",
                    "CmdPath":"/usr/bin/python",
                    "TimeoutMilliSeconds":1000,
                    "IgnoreError":false,
                    "Environments":[
                        "DEBUG=true"
                    ],
                    "IntervalMs":5000
                }
            }
        ]
    }

    Parameter

    Type

    Required

    Description

    type

    string

    Yes

    The type of the data source. Set the value to input_command.

    ScriptType

    string

    Yes

    The type of the script. Valid values: bash, shell, python2, and python3.

    User

    string

    Yes

    The name of the user that is used to execute the script. Only non-root users are supported.

    Note
    • Make sure that the specified user name exists on the server.

    • We recommend that you grant the user only the rwx permissions on the required directories or files.

    ScriptContent

    string

    Yes

    The content of the script. The content can be Base64-encoded or plaintext. The content must be within 524,288 bytes in length.

    ContentEncoding

    string

    No

    The text format of the script content. This parameter is optional. Valid values:

    • PlainText (default)

    • Base64

    LineSplitSep

    string

    No

    The delimiter that is used to separate the output of the script. If you leave this parameter empty, the output of the script is returned as a single data record.

    CmdPath

    string

    No

    The path to the script. If you leave this parameter empty, the default path is used. Default paths:

    • bash: /usr/bin/bash

    • shell: /usr/bin/sh

    • python2: /usr/bin/python2

    • python3: /usr/bin/python3

    TimeoutMilliSeconds

    string

    No

    The timeout period for script execution. Default value: 3000. Unit: milliseconds.

    IgnoreError

    Bool

    No

    Specifies whether to ignore error logs that are generated when Logtail fails to execute the script. Default value: false. The value false indicates that error logs are not ignored.

    Environments

    []string

    No

    The environment variable. By default, the value of os.Environ() is used. If you configure this parameter, the value of this parameter is used together with the value of os.Environ().

    IntervalMs

    int

    Yes

    The intervals at which logs are collected or the script is executed. Default value: 5000. Unit: milliseconds.

  7. Preview data, configure indexes, and then click Next.

    By default, full-text indexing is enabled in Simple Log Service. You can configure field indexes based on collected logs in manual or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. Simple Log Service automatically creates field indexes. For more information, see Create indexes.

    Important

    If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.

  8. Click Log Query. Then, you are redirected to the query and analysis page of your Logstore.

    You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

Example

Obtain the system time by using a python2 script.

  • Logtail configuration

    {
        "inputs":[
            {
                "type":"input_command",
                "detail":{
                    "ScriptType":"python2",
                    "User":"test",
                    "ScriptContent":"print(\"test input_command 0\")\nprint(\"test input_command 1\")\nprint(\"test input_command 2\")\nprint(\"test input_command 3\")\nprint(\"test input_command 4\")\nprint(\"test input_command 5\")\nprint(\"test input_command 6\")\nprint(\"test input_command 7\")\nprint(\"test input_command 8\")\nprint(\"test input_command 9\")\n",
                    "ContentEncoding":"PlainText",
                    "CmdPath":"/usr/bin/python",
                    "TimeoutMilliSeconds":1000,
                    "IgnoreError":false,
                    "Environments":[
                        "DEBUG=true"
                    ],
                    "IntervalMs":5000
                }
            }
        ]
    }
  • Collected log

    content:"Mon Aug  7 02:17:52 UTC 2023"
    script_md5:"a9564ebc3289b7a14551baf8ad5ec60a"
    __pack_meta__:"1|MTY4OTI1MzExNTU2NTE2MzQ1NA==|2|1"
    __topic__:""
    __source__:"172.17.0.2"
    __tag__:__pack_id__:"D7B7F988051A3019-0"
    __tag__:__hostname__:"bc6c7472f257"
    __tag__:__client_ip__:"8.218.101.91"
    __tag__:__receive_time__:"1691374673"
    __time__:"1691374672"
  • Field description

    Field

    Description

    content

    The script execution log that is collected.

    script_md5

    The MD5 hash value of the script content.

Troubleshooting

If no data is displayed on the preview page or query page after logs are collected by using Logtail, you can troubleshoot the errors based on the instructions that are provided in What do I do if errors occur when I use Logtail to collect logs?