All Products
Search

This Product

    Simple Log Service:Consume data with Logstash

    Document Center

    Simple Log Service:Consume data with Logstash

    Last Updated:Jun 20, 2026

    You can use Logstash to consume data from Simple Log Service. Configure the Simple Log Service Simple Log Service to retrieve data from Simple Log Service and write it to other systems, such as Kafka or HDFS.

    Features

    • Distributed consumption: Configure multiple servers to consume data from a single Logstore at the same time.

    • High performance: Based on the Java consumer group implementation, a single core can consume uncompressed data at up to 20 MB/s.

    • High reliability: The consumption progress is saved on the server. If an exception occurs, consumption automatically resumes from the last checkpoint.

    • Automatic load balancing: Shards are automatically allocated based on the number of consumers. The load is automatically rebalanced if consumers are added or removed.

    Procedure

    Download the Logstash installation package for your operating system from Logstash.

    This topic uses a Linux system as an example:

    1. Install Logstash. For more information, see Logstash.

      1. Download and install the public signing key.

        sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

      2. In the /etc/yum.repos.d/ directory, create a file with a .repo extension, such as logstash.repo, and add the following content:

        [logstash-9.x]
name=Elastic repository for 9.x packages
baseurl=https://artifacts.elastic.co/packages/9.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

      3. Download and install Logstash.

        sudo yum install logstash

    2. Install the input plugin.

      1. Download the input plugin from logstash-input-sls.

      2. Install the input plugin.

        /usr/share/logstash/bin/logstash-plugin install logstash-input-sls.zip
        Note

        For information about potential installation failures and solutions, see Plugin installation and configuration.

    3. Create the logstash user. Logstash must run as a non-root user.

      1. Create the logstash user.

        sudo adduser --system --no-create-home --group logstash

      2. Set the permissions for the logstash user. Ensure that the related Logstash directories, such as /usr/share/logstash, /etc/logstash, and /var/log/logstash, are owned by the logstash user.

        sudo chown -R logstash:logstash /usr/share/logstash /etc/logstash /var/log/logstash

      3. Verify that the logstash user is created.

        id logstash

        The output shows the user ID (UID) and group ID (GID) for the logstash user, which confirms that the user was created successfully.

        [root@iZxxx ~]# id logstash
uid=995(logstash) gid=992(logstash) groups=992(logstash)

    4. Start Logstash as the logstash user.

      1. In the /etc/logstash directory, create a file with the .conf extension. This topic uses logstash-sample.conf as an example.

      2. In the logstash-sample.conf file, add the sample code and start Logstash as the logstash user.

        sudo -u logstash /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-sample.conf

        The following example configures Logstash to consume data from a Logstore and print it to standard output. The following table describes the parameters.

        Parameters

        Sample code

        input {
  logservice{
  endpoint => "your project endpoint"
  access_id => "your_accesskey_id"
  access_key => "your_accesskey_secret"
  project => "your project name"
  logstore => "your logstore name"
  consumer_group => "consumer group name"
  consumer_name => "consumer name"
  position => "end"
  checkpoint_second => 30
  include_meta => true
  consumer_name_with_ip => true
  }
}

output {
  stdout {}
}

        endpointstring (Required)

        The service endpoint for the SLS Project. For more information, see Service endpoints.

        access_idstring (Required)

        The AccessKey ID for your Alibaba Cloud account. This key must have permissions to manage the consumer group. For more information, see Grant permissions to consume data from a Logstore.

        access_keystring (Required)

        The AccessKey Secret for your Alibaba Cloud account. This key must have permissions to manage the consumer group. For more information, see Grant permissions to consume data from a Logstore.

        projectstring (Required)

        The name of the SLS Project.

        logstorestring (Required)

        The name of the SLS Logstore.

        consumer_groupstring (Required)

        The name of the consumer group.

        consumer_namestring (Required)

        The name of the consumer. The name must be unique within a consumer group.

        positionstring (Required)

        The consumption start position.

        • begin: Start consumption from the first entry in the Logstore.

        • end: Start consumption from the current time.

        • yyyy-MM-dd HH:mm:ss: Start consumption from a specific point in time.

        checkpoint_secondnumber (Optional)

        The interval, in seconds, for saving a checkpoint. The recommended range is 10 to 60. The value must be at least 10. Default: 30.

        include_metaboolean (Optional)

        Specifies whether to include metadata, such as source, time, tag, and topic. Default: true.

        consumer_name_with_ipboolean (Optional)

        Specifies whether to append the IP address to the consumer name. This parameter must be set to true for distributed consumption. Default: true.