Simple Log Service:Collect script execution logs

Prerequisites

Logtail V1.7.1 or later is installed on your Linux server. For more information, see Install Logtail on a Linux server.

Description

You can specify a user and a script in a Logtail configuration. Logtail executes the script as the user to obtain script execution logs.

• You can specify one of the following types of scripts: bash, shell, python2, and python3.

• You can specify a user to execute the script. This way, you can achieve user-based access control on resources that can be accessed by the script.

• If the script execution times out, Logtail automatically stops executing the script. Logtail does not collect the script execution logs.

Procedure

1. Log on to the Simple Log Service console.

2. In the Import Data section, select Custom Data Plug-in.

3. Select a project and a Logstore. Then, click Next.

4. Create a machine group.

   • If a machine group is available, click Use Existing Machine Groups.

   • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.

     1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.

        For more information, see Install Logtail on ECS instances.

        Important

        If your server is an ECS instance that belongs to a different Alibaba Cloud account from Simple Log Service, a server in a data center, or a server from a third-party cloud service provider, you must manually install Logtail on the server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server. After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.

     2. After Logtail is installed, click Complete Installation.

     3. In the Create Machine Group step, configure the Name parameter and click Next.

        Simple Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.

5. Select the machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.

   Important

   If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, troubleshoot the issue by following the instructions provided in How do I troubleshoot an error that is related to a Logtail machine group in a host environment?

6. In the Specify Data Source step, configure the Config Name and Plug-in Config parameters. Then, click Next.

   • inputs is required and is used to configure the data collection settings for the Logtail configuration. You must configure inputs based on your data source.

     Important

     You can configure only one type of data source in the inputs field.

   • processors is optional and is used to configure the data processing settings for the Logtail configuration. You can specify one or more processing methods.

     If your logs cannot be parsed based only on the setting of inputs, you can configure processors in the Plug-in Config field to add plug-ins for data processing. For example, you can extract fields, extract log time, mask data, and filter logs. For more information, see Overview.

   {
       "inputs":[
           {
               "type":"input_command",
               "detail":{
                   "ScriptType":"python2",
                   "User":"test",
                   "ScriptContent":"print(\"test input_command 0\")\nprint(\"test input_command 1\")\nprint(\"test input_command 2\")\nprint(\"test input_command 3\")\nprint(\"test input_command 4\")\nprint(\"test input_command 5\")\nprint(\"test input_command 6\")\nprint(\"test input_command 7\")\nprint(\"test input_command 8\")\nprint(\"test input_command 9\")\n",
                   "ContentEncoding":"PlainText",
                   "CmdPath":"/usr/bin/python",
                   "TimeoutMilliSeconds":1000,
                   "IgnoreError":false,
                   "Environments":[
                       "DEBUG=true"
                   ],
                   "IntervalMs":5000
               }
           }
       ]
   }

   Parameter	Type	Required	Description
   type	string	Yes	The type of the data source. Set the value to input_command.
   ScriptType	string	Yes	The type of the script. Valid values: bash, shell, python2, and python3.
   User	string	Yes	The name of the user that is used to execute the script. Only non-root users are supported. Note Make sure that the specified user name exists on the server. We recommend that you grant the user only the rwx permissions on the required directories or files.
   ScriptContent	string	Yes	The content of the script. The content can be Base64-encoded or plaintext. The content must be within 524,288 bytes in length.
   ContentEncoding	string	No	The text format of the script content. This parameter is optional. Valid values: PlainText (default) Base64
   LineSplitSep	string	No	The delimiter that is used to separate the output of the script. If you leave this parameter empty, the output of the script is returned as a single data record.
   CmdPath	string	No	The path to the script. If you leave this parameter empty, the default path is used. Default paths: bash: /usr/bin/bash shell: /usr/bin/sh python2: /usr/bin/python2 python3: /usr/bin/python3
   TimeoutMilliSeconds	string	No	The timeout period for script execution. Default value: 3000. Unit: milliseconds.
   IgnoreError	Bool	No	Specifies whether to ignore error logs that are generated when Logtail fails to execute the script. Default value: false. The value false indicates that error logs are not ignored.
   Environments	[]string	No	The environment variable. By default, the value of os.Environ() is used. If you configure this parameter, the value of this parameter is used together with the value of os.Environ().
   IntervalMs	int	Yes	The intervals at which logs are collected or the script is executed. Default value: 5000. Unit: milliseconds.

7. Preview data, configure indexes, and then click Next.

   By default, full-text indexing is enabled in Simple Log Service. You can configure field indexes based on collected logs in manual or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. Simple Log Service automatically creates field indexes. For more information, see Create indexes.

   Important

   If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.

8. Click Log Query. Then, you are redirected to the query and analysis page of your Logstore.

   You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

Example

Obtain the system time by using a python2 script.

• Logtail configuration

  {
      "inputs":[
          {
              "type":"input_command",
              "detail":{
                  "ScriptType":"python2",
                  "User":"test",
                  "ScriptContent":"print(\"test input_command 0\")\nprint(\"test input_command 1\")\nprint(\"test input_command 2\")\nprint(\"test input_command 3\")\nprint(\"test input_command 4\")\nprint(\"test input_command 5\")\nprint(\"test input_command 6\")\nprint(\"test input_command 7\")\nprint(\"test input_command 8\")\nprint(\"test input_command 9\")\n",
                  "ContentEncoding":"PlainText",
                  "CmdPath":"/usr/bin/python",
                  "TimeoutMilliSeconds":1000,
                  "IgnoreError":false,
                  "Environments":[
                      "DEBUG=true"
                  ],
                  "IntervalMs":5000
              }
          }
      ]
  }

• Collected log

  content:"Mon Aug  7 02:17:52 UTC 2023"
  script_md5:"a9564ebc3289b7a14551baf8ad5ec60a"
  __pack_meta__:"1|MTY4OTI1MzExNTU2NTE2MzQ1NA==|2|1"
  __topic__:""
  __source__:"172.17.0.2"
  __tag__:__pack_id__:"D7B7F988051A3019-0"
  __tag__:__hostname__:"bc6c7472f257"
  __tag__:__client_ip__:"8.218.101.91"
  __tag__:__receive_time__:"1691374673"
  __time__:"1691374672"

• Field description

  Field	Description
  content	The script execution log that is collected.
  script_md5	The MD5 hash value of the script content.