Logstash is an open source data ingestion engine that is capable of dynamically ingesting data from data sources. With Logstash, you can use custom rules to filter data ingested form different data sources and then output the data to the target service. This topic describes how to deploy Logstash on Elastic Compute Service (ECS), and provides an example to describe how to migrate data from Logstash to Alibaba Cloud Elasticsearch (ES).

Prerequisites

Before you deploy and use Logstash, you must create an Alibaba Cloud ECS instance and Elasticsearch instance, and configure them.

Create and configure an Alibaba Cloud Elasticsearch instance
  1. Create an Alibaba Cloud Elasticsearch instance.

    In this example, the version of the Elasticsearch instance is V6.7.0. The following figure shows the configuration of the Elasticsearch instance.

    Elasticsearch configuration
  2. Log on to the Alibaba Cloud Elasticsearch console, and enable auto-indexing.
  3. Log on to the Kibana console of the Elasticsearch instance, and add a role that is granted the read and write permission: logstash-*.
    1. Log on to the Kibana console.
    2. Choose Management > Roles > Create role.
      Create a role
    3. On the Create role page, set the relevant parameters.
      Parameters
      Parameter Description
      Role name The name of the role.
      Indices The index files to be migrated. Enter logstash-*.
      Privileges The permissions granted to the role. Add the read, write, create, delete, and create_index permissions.
      Granted fields The fields that the role is authorized to access. This parameter is optional. In this example, enter *.
    4. Click Create role to create the role.
Create and configure an Alibaba Cloud ECS instance
  1. Create an ECS instance. Make sure that the ECS instance can access the Logstash instance and Elasticsearch instance. You can also use a purchased ECS instance that meets the requirements.

    The following figure shows the configuration of the ECS instance.

    ECS configuration
    Note We recommend that you purchase an ECS instance in the same zone and VPC network as your Elasticsearch instance. You can also purchase a classic network-connected ECS instance. However, you must first make sure that the classic network is connected to the VPC network of the Elasticsearch instance.
  2. Install the JDK on the ECS instance. The JDK version must be V1.8 or later.

    For more information about how to install the JDK, see Install the JDK.

Install Logstash

  1. Download Logstash V6.7.0.
    Visit the official Elasticsearch website, and download Logstash of the same version as your Elasticsearch version. We recommend that you download Logstash V6.7.0.
    wget https://artifacts.elastic.co/downloads/logstash/logstash-6.7.0.tar.gz
  2. Extract the Logstash package.
    tar -xzvf logstash-6.7.0.tar.gz

Use Logstash to synchronize the incremental data.

  1. Connect to the ECS instance, and then switch to the Logstash directory.
    cd logstash-6.7.0
  2. Create a .conf file named test.
    touch test.conf
  3. Configure the test.conf file. The following is an example:
    input {
        file {
            path => "/your/file/path/xxx"
            }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => ["http://instanceId.elasticsearch.aliyuncs.com:9200"]
        user => "user-name"
        password => "logstash-password"
      }
    }
    Parameter Description
    path The path of the log files. In this example, the path is /var/log/meaasges.
    hosts The endpoint of the Elasticsearch instance. Replace instanceId with the ID of the Elasticsearch instance. You can check the instance ID on the Basic Information page of the Elasticsearch instance. Example: http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200.
    user The username of the Elasticsearch instance. The default username is elastic. If you want to use a custom account, you must first create a role for the account and grant the required permissions to the role. For more information, see Create a user and Create a role.
    Notice The username must be enclosed in a pair of straight quotation marks ("). This helps you avoid errors incurred by special characters in the username when you launch Logstash.
    password The password of the Elasticsearch instance. The password is specified when you create the Elasticsearch instance. You can also change the password after the Elasticsearch instance is created.
    Notice The password must be enclosed in a pair of straight quotation marks ("). This helps you avoid errors incurred by special characters in the password when you launch Logstash.

    Logstash provides a variety of input, filter, and output plug-ins. These plug-ins can help you easily ingest, transform, and output data. For more information, see Structure of a config file.

  4. Run the logstash command.
    Follow step 3 to configure the .conf file, and then run the logstash command.
    bin/logstash -f test.conf

    After the command is executed, the system automatically uses Logstash to ingest new content from the log files and output the content to the Elasticsearch instance. Logstash will index any changes in the log files to the Elasticsearch instance.

  5. Verify the result.

    Before you verify the synchronization result, you must create a role, and then authorize the role to manage the logstash-* index. The required permissions are read, write, create, delete, and create_index. Fore more information, see Create a role.

    1. Log on to the Kibana console of the Elasticsearch instance.
    2. In the left-side navigation pane, choose Dev Tools.
    3. On the Console tab of the Dev Tools page, run the following command:
      GET /logstash-*/_search

      After the command is executed, the following result is returned.

      Returned result

Monitor Logstash nodes

Follow these steps to monitor Logstash nodes and collect monitoring data.
  1. Switch to the config folder in the Logstash directory.
    cd /logstash-6.7.0/config
  2. Configure the logstash.yml file.

    Open the logstash.yml file.

    vim logstash.yml

    Delete the annotations of the following parameters in the logstash.yml file, and assign values to these parameters.

    Configure X-Pack monitoring
    Parameter Description
    xpack.monitoring.enable Set this parameter to true. The default value is false.
    xpack.monitoring.elasticsearch.username Create a user to monitor Logstash. For more information, see Create a user.
    Notice You can also use the elastic account. However, we recommend that you do not use the elastic account in a production environment because this may pose system security risks.
    xpack.monitoring.elasticsearch.password The password of the user that is used to monitor Logstash.
    xpack.monitoring.elasticsearch.hosts The endpoint of the Elasticsearch instance. Example: http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200.
  3. Return to the directory of Logstash, and then launch the Logstash service.
    cd ../
    bin/logstash -f test.conf

    After Logstash is launched, the following result is returned.

    Logstash launched
  4. Log on to the Kibana console of the Elasticsearch instance. In the left-side navigation pane, choose Monitoring to view the Logstash monitoring data.
    View Logstash monitoring data

Create a user

This section describes how to create a user to monitor Logstash from the CLI or Kibana console.
Notice By default, Alibaba Cloud Elasticsearch disables the logstash_system user and you are not allowed to create the logstash_system user. Therefore, you need to create a user that assumes the logstash_system role.

Create a user from the CLI

Connect to the ECS instance and run the following command to create a user.
curl -u elastic:es-password -XPOST http://instanceId.elasticsearch.aliyuncs.com:9200/_xpack/security/user/logstash_system_monitor -d '{"password" : "logstash-monitor-password","roles" : ["logstash_system"],"full_name" : "your full name"}'
Parameter Description
es-password The password of the Elasticsearch instance. This password is also used to log on to the Kibana console.
instanceId The ID of the Elasticsearch instance. You can check the instance ID on the Basic Information page of the Elasticsearch instance.
logstash-monitor-password The password of the logstash_system_monitor user.
your full name The full name of the user.

After the user is created, the following result is returned.

User created

Create a user from the Kibana console

  1. Log on to the Kibana console.
  2. Choose Management > Users > Create new user.
    Create a user
  3. On the New user page, enter the user information.
    New user
    Parameter Description
    Username The name of the user. You can enter a custom name. In this section, user logstash_system_monitor is created to monitor Logstash.
    Password The password of the user.
    Confirm password Enter the password again to confirm.
    Full name The full name of the user. This parameter is optional.
    Email address The email address of the user. This parameter is optional.
    Roles The role that the user assumes. Specify the logstash_system role.
  4. Click Create user to create the user.

FAQ

  • Q: Why must I enable auto-indexing for the Elasticsearch instance before I use Logstash to output data to Elasticsearch?

    A: To guarantee data security, Alibaba Cloud Elasticsearch disables auto-indexing by default.

    Instead of calling the Create index operation, Logstash submits data to Elasticsearch. Elasticsearch then automatically indexes the data. Therefore, before you use Logstash to output data to Elasticsearch, you must enable auto-indexing for your Elasticsearch instance.Enable auto-indexing
    Notice After you enable auto-indexing and confirm the operation, the system will restart the Elasticsearch instance. Make sure that your workloads running on the instance are not adversely affected before you confirm the operation.
  • Q: What can I do if the system prompts that I do not have the permission to create indexes?

    System prompts

    A: You need to check the role of the Elasticsearch account that is used to receive the data. Make sure that the role has the write, delete, and create_index permissions.

  • Q: How do I resolve the error of insufficient memory?

    Insufficient memory

    A: By default, 1 GB of memory is allocated to Logstash. If your purchased ECS instance does not have sufficient memory for Logstash, modify config/jvm.options to decrease the amount of memory allocated to Logstash.

  • Q: How do I resolve the following error that occurs when I run the Logstash command?

    Error

    A: You must verify that the username and password in the test.conf file do not contain special characters. If the username or password contains special characters, it must be enclosed in a pair of straight quotation marks (").