This topic describes how to configure Logtail in the Log Service console to collect data from Beats and Logstash.

Prerequisites

  • Logtail is installed on the server that you use to collect data from Beats and Logstash. For more information, see Install Logtail in Linux or Install Logtail in Windows.
    Note Servers that run Linux support Logtail 0.16.9 or later. Servers that run Windows support Logtail 1.0.0.8 or later.
  • Data is collected by using Logstash or Beats.
    • For more information about how to collect data from Logstash, visit Logstash-Lumberjack-Output.
    • For more information about how to collect data from Beats, visit Beats-Lumberjack-Output.
      The procedure in this topic describes how to use Packetbeat to collect data transmitted on the local network, and use the Logtail Lumberjack plug-in to upload the data to Log Service. Data collected by using Packetbeat is sent to Logstash, as shown in the following sample script:
      output.logstash:
        hosts: ["127.0.0.1:5044"]

Background information

Logstash and Beats (such as MetricBeat, PacketBeat, Winlogbeat, Auditbeat, Filebeat, and Heartbeat) support the Lumberjack protocol. Therefore, Logtail can use the protocol to upload data that is collected by Beats and Logstash to Log Service.
Note
  • You can configure multiple Lumberjack plug-ins, but these plug-ins cannot listen on the same port.
  • Lumberjack plug-ins support SSL. Data uploaded to Log Service from Logstash must be encrypted by using SSL.

Procedure

  1. Log on to the Log Service console.
  2. In the Import Data section, select Custom Data Plug-in.
  3. In the Specify Logstore step, select the target project and Logstore, and click Next.
    You can also click Create Now to create a project and a Logstore. For more information, see Step 1: Create a project and a Logstore.
  4. In the Create Machine Group step, create a machine group.
    • If a machine group is available, click Using Existing Machine Groups.
    • This section uses ECS instances as an example to describe how to create a machine group. To create a machine group, perform the following steps:
      1. Install Logtail on ECS instances. For more information, see Install Logtail on ECS instances.

        If Logtail is installed on the ECS instances, click Complete Installation.

        Note If you need to collect logs from user-created clusters or servers of third-party cloud service providers, you must install Logtail on these servers. For more information, see Install Logtail in Linux or Install Logtail in Windows.
      2. After the installation is complete, click Complete Installation.
      3. On the page that appears, specify the parameters for the machine group. For more information, see Create an IP address-based machine group or Create a custom ID-based machine group.
  5. In the Machine Group Settings step, apply the configurations to the machine group.
    Select the created machine group and move the group from Source Server Groups to Applied Server Groups.
  6. In the Specify Data Source step, set the Config Name and Plug-in Config parameters.
    • inputs: Required. The Logtail configurations for log collection.
      Note You can configure only one type of data source in the inputs field.
    • processors: Optional. The Logtail configurations for data processing. You can configure one or more processing methods in the processors field. For more information, see Process data.

    Data from Beats and Logstash is in the JSON format. processor_anchor is configured to expand the JSON-formatted data.

    {
      "inputs": [
        {
          "detail": {
            "BindAddress": "0.0.0.0:5044"
          },
          "type": "service_lumberjack"
        }
      ],
      "processors": [
        {
          "detail": {
            "Anchors": [
              {
                "ExpondJson": true,
                "FieldType": "json",
                "Start": "",
                "Stop": ""
              }
            ],
            "SourceKey": "content"
          },
          "type": "processor_anchor"
        }
      ]
    }
    						
    Parameter Type Required Description
    type String Yes The type of the data source. Set the value to service_lumberjack.
    BindAddress String No The IP address and port of the server to which data can be sent by using the Lumberjack protocol. Default value: 127.0.0.1:5044. To enable access from other hosts in the LAN by using the Lumberjack protocol, set the value to 0.0.0.0:5044.
    V1 Boolean No Specifies whether to use the Lumberjack protocol v1. Default value: false. Logstash supports the Lumberjack protocol v1.
    V2 Boolean No Specifies whether to use the Lumberjack protocol v2. Default value: true. Beats support the Lumberjack protocol v2.
    SSLCA String No The path of the Certificate Authority that issues the signature certificate. Default value: null. If you use a self-signed certificate, you do not need to specify the parameter.
    SSLCert String No The path of the certificate. Default value: null.
    SSLKey String No The path of the private key that corresponds to the certificate. Default value: null.
    InsecureSkipVerify Boolean No Specifies whether to skip the SSL security check. Default value: false. This value indicates the SSL security check is performed.
  7. In the Configure Query and Analysis step, configure the indexes.
    Indexes are configured by default. You can re-configure the indexes based on your business requirements. For more information, see Enable and configure the index feature for a Logstore.
    Note
    • You must configure Full Text Index or Field Search. If you configure both of them, the settings of Field Search are applied.
    • If the data type of index is long or double, the Case Sensitive and Delimiter settings are unavailable.

What to do next

After Logtail uploads data to Log Service, you can view the data in the Log Service console. The following content is the sample data uploaded to Log Service.

_@metadata_beat:  packetbeat
_@metadata_type:  doc
_@metadata_version:  6.2.4
_@timestamp:  2018-06-05T03:58:42.470Z
__source__:  **. **. **.**
__tag__:__hostname__:  *******
__topic__:  
_beat_hostname:  bdbe0b8d53a4
_beat_name:  bdbe0b8d53a4
_beat_version:  6.2.4
_bytes_in:  56
_bytes_out:  56
_client_ip:  192.168.5.2
_icmp_request_code:  0
_icmp_request_message:  EchoRequest(0)
_icmp_request_type:  8
_icmp_response_code:  0
_icmp_response_message:  EchoReply(0)
_icmp_response_type:  0
_icmp_version:  4
_ip:  127.0.0.1
_path:  127.0.0.1
_responsetime:  0
_status:  OK
_type:  icmp