All Products
Search
Document Center

Simple Log Service:Use Logstash to consume log data

Last Updated:Aug 24, 2023

Simple Log Service allows you to use Logstash to consume log data. You can configure the Logstash input plug-in to read log data from Simple Log Service and write the data to other systems, such as Kafka and Hadoop Distributed File System (HDFS).

Features

  • Distributed collaborative consumption: Multiple servers can be configured to consume log data from a Logstore at the same time.

  • High performance: If you use a Java consumer group, the consumption speed of a single-core CPU can reach 20 MB/s.

  • High reliability: Simple Log Service saves consumption checkpoints. This mechanism resumes log consumption from the last checkpoint after a consumption exception is resolved.

  • Automatic load balancing: Shards are automatically allocated based on the number of consumers in a consumer group. If a consumer joins or leaves the consumer group, shards are automatically reallocated.

Procedure

  1. Install Logstash.

    1. Download the installation package.

    2. Decompress the downloaded package to the specified directory.

  2. Install the Logstash input plug-in.

    1. Download the Logstash input plug-in. To download the plug-in, visit logstash-input-sls.

    2. Install the Logstash input plug-in.

      logstash-plugin install logstash-input-sls
      Note

      For information about the causes of installation failures and solutions, see Plug-in installation and configuration.

  3. Start Logstash.

    logstash -f logstash.conf

    The following table describes the parameters.

    Parameter

    Type

    Required

    Description

    endpoint

    string

    Yes

    The endpoint for the region where the Simple Log Service project resides. For more information, see Endpoints.

    access_id

    string

    Yes

    The AccessKey ID of the Alibaba Cloud account or RAM user that is used to access the consumer group. For more information, see Use custom policies to grant permissions to a RAM user.

    access_key

    string

    Yes

    The AccessKey secret of the Alibaba Cloud account or RAM user that is used to access the consumer group. For more information, see Use custom policies to grant permissions to a RAM user.

    project

    string

    Yes

    The name of the Simple Log Service project.

    logstore

    string

    Yes

    The name of the Simple Log Service Logstore.

    consumer_group

    string

    Yes

    The name of the consumer group.

    consumer_name

    string

    Yes

    The name of the consumer. The name must be unique in a consumer group.

    position

    string

    Yes

    The position where data consumption starts. Valid values:

    • begin: consumes data from the first log that is written to the Logstore.

    • end: consumes data from the current time.

    • yyyy-MM-dd HH:mm:ss: consumes data from the specified point in time.

    checkpoint_second

    number

    No

    The interval at which checkpoints are recorded. We recommend that you set the interval to a value between 10 seconds and 60 seconds. Minimum value: 10. Default value: 30. Unit: seconds.

    include_meta

    boolean

    No

    Specifies whether the logs contain metadata, such as the log source, time, tags, and topic fields. Default value: true.

    consumer_name_with_ip

    boolean

    No

    Specifies whether to include an IP address in a consumer name. Default value: true. You must set this parameter to true if distributed collaborative consumption is applied.

Examples

The following code provide an example on how to configure Logstash to consume log data from a Logstore and print the data into standard output (stdout):

Important

We recommend that you do not save the AccessKey ID or AccessKey secret in your project code. Otherwise, the AccessKey pair may be leaked, and the security of all resources within your account may be compromised. We recommend that you configure the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables. For more information, see Configure environment variables.

input {
  logservice{
  endpoint => "your project endpoint"
  // Configure environment variables. In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
  access_id => process.env.ALIBABA_CLOUD_ACCESS_KEY_ID
  access_key => process.env.ALIBABA_CLOUD_ACCESS_KEY_SECRET
  project => "your project name"
  logstore => "your logstore name"
  consumer_group => "consumer group name"
  consumer_name => "consumer name"
  position => "end"
  checkpoint_second => 30
  include_meta => true
  consumer_name_with_ip => true
  }
}

output {
  stdout {}
}