This topic describes how to configure Logtail in the Log Service console to collect data from Beats and Logstash.


  • Logtail is installed on the server that you use to collect data from Beats and Logstash. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.
    Note Servers that run Linux support Logtail 0.16.9 or later. Servers that run Windows support Logtail or later.
  • Data is collected by using Logstash or Beats.
    • For more information about how to collect data from Logstash, visit Logstash-Lumberjack-Output.
    • For more information about how to collect data from Beats, visit Beats-Lumberjack-Output.
      The procedure in this topic describes how to use Packetbeat to collect data transmitted on the local network, and use the Logtail Lumberjack plug-in to upload the data to Log Service. Data collected by using Packetbeat is sent to Logstash, as shown in the following sample script:
        hosts: [""]

Background information

Logstash and Beats (such as MetricBeat, PacketBeat, Winlogbeat, Auditbeat, Filebeat, and Heartbeat) support the Lumberjack protocol. Therefore, Logtail can use the protocol to upload data that is collected by Beats and Logstash to Log Service.
  • You can configure multiple Lumberjack plug-ins, but these plug-ins cannot listen on the same port.
  • Lumberjack plug-ins support SSL. Data uploaded to Log Service from Logstash must be encrypted by using SSL.


  1. Log on to the Log Service console.
  2. In the Import Data section, click Custom Data Plug-in.
  3. Select the project and Logstore. Then, click Next.
  4. Create a machine group.
    • If a machine group is available, click Use Existing Machine Groups.
    • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
      1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.

        For more information, see Install Logtail on ECS instances.

        Important If you want to collect logs from an ECS instance that belongs to a different Alibaba Cloud account, a server in an on-premises data center, or a server of a third-party cloud service provider, you must manually install Logtail. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.

        After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.

      2. After Logtail is installed, click Complete Installation.
      3. In the Create Machine Group step, configure the Name parameter and click Next.

        Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.

  5. Select the new machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.
    Important If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
  6. In the Specify Data Source step, set the Config Name and Plug-in Config parameters.
    • inputs specifies the collection configurations of your data source. This parameter is required.
      Important You can specify only one type of data source in the inputs parameter.
    • processors specifies the processing configurations that are used to parse data. You can extract fields, extract log time, desensitize data, and filter logs. This parameter is optional. You can specify one or more processing methods. For more information, see Overview.

    Data from Beats and Logstash is in the JSON format. processor_anchor is configured to expand the JSON-formatted data.

      "inputs": [
          "detail": {
            "BindAddress": ""
          "type": "service_lumberjack"
      "processors": [
          "detail": {
            "Anchors": [
                "ExpondJson": true,
                "FieldType": "json",
                "Start": "",
                "Stop": ""
            "SourceKey": "content"
          "type": "processor_anchor"
    typeStringYesThe type of the data source. Set the value to service_lumberjack.
    BindAddressStringNoThe IP address and port of the server to which data can be sent by using the Lumberjack protocol. Default value: To enable access from other hosts in the LAN by using the Lumberjack protocol, set the value to
    V1BooleanNoSpecifies whether to use the Lumberjack protocol v1. Default value: false. Logstash supports the Lumberjack protocol v1.
    V2BooleanNoSpecifies whether to use the Lumberjack protocol v2. Default value: true. Beats support the Lumberjack protocol v2.
    SSLCAStringNoThe path of the Certificate Authority that issues the signature certificate. Default value: null. If you use a self-signed certificate, you do not need to specify the parameter.
    SSLCertStringNoThe path of the certificate. Default value: null.
    SSLKeyStringNoThe path of the private key that corresponds to the certificate. Default value: null.
    InsecureSkipVerifyBooleanNoSpecifies whether to skip the SSL security check. Default value: false. This value indicates the SSL security check is performed.
  7. Preview data, configure indexes, and then click Next.
    By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.
    Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
  8. Click Log Query. You are redirected to the query and analysis page of your Logstore.
    You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.


If no data is displayed on the preview page or query page after logs are collected by using Logtail, you can troubleshoot the errors based on the instructions that are provided in What do I do if errors occur when I use Logtail to collect logs?

What to do next

After Logtail uploads data to Log Service, you can view the data in the Log Service console. The following content is the sample data uploaded to Log Service.

_@metadata_beat:  packetbeat
_@metadata_type:  doc
_@metadata_version:  6.2.4
_@timestamp:  2018-06-05T03:58:42.470Z
__source__:  **. **. **.**
__tag__:__hostname__:  *******
_beat_hostname:  bdbe0b8d53a4
_beat_name:  bdbe0b8d53a4
_beat_version:  6.2.4
_bytes_in:  56
_bytes_out:  56
_icmp_request_code:  0
_icmp_request_message:  EchoRequest(0)
_icmp_request_type:  8
_icmp_response_code:  0
_icmp_response_message:  EchoReply(0)
_icmp_response_type:  0
_icmp_version:  4
_responsetime:  0
_status:  OK
_type:  icmp