All Products
Search
Document Center

Elasticsearch:Use a self-managed Logstash cluster to migrate data to an Alibaba Cloud Elasticsearch cluster

Last Updated:Jan 12, 2024

Logstash is an open source data collection engine that can filter data and transmit data in real time. If you do not have high requirements for real-time performance of data migration, you can use Logstash to migrate data. This topic describes how to use Alibaba Cloud Elastic Compute Service (ECS) to deploy a self-managed Logstash cluster and how to use the Logstash cluster to migrate data in a log file to an Alibaba Cloud Elasticsearch cluster.

Prerequisites

  • An Alibaba Cloud Elasticsearch cluster is created, and the Auto Indexing feature is enabled for the cluster.

    Note
  • An ECS instance is created. The ECS instance can connect to both the Elasticsearch cluster and the self-managed Logstash cluster. For more information, see Create an instance by using the wizard.

    Note

    We recommend that you create an ECS instance that resides in the same region, zone, and virtual private cloud (VPC) as the Elasticsearch cluster.

  • The JDK is installed on the ECS instance. For more information, see Install the JDK.

    Note

    The JDK must be of Temurin 17 or later. For more information about the compatibility between Java JDK versions and Logstash versions, see Compatibility between Logstash and JVM.

  • Data to be migrated is prepared. In this example, ECS log data is used.

Procedure

Step 1: (Optional) Create a role and a user

You need to use an Elasticsearch user when you connect a self-managed Logstash cluster to an Alibaba Cloud Elasticsearch cluster or view Logstash monitoring data in the Kibana console. You can use the elastic user or a custom user.

Note

In actual business, we recommend that you use a custom user. Before you use a custom user, you must assign a role with the required permissions to the user. If you use the elastic user in actual business, system security cannot be ensured.

Create a role

In this example, a role that is granted the read and write permissions on logstash-* indexes is created.

  1. Log on to the Kibana console of the Elasticsearch cluster.

    For more information, see Log on to the Kibana console.

  2. Click the image icon in the upper-left corner of the page that appears, and choose Management > Stack Management.

  3. In the left-side navigation pane, choose Security > Roles.

  4. On the Roles page, click Create role in the upper-right corner. On the Create role page, configure the parameters.

    The following table describes some parameters that are displayed.

    Parameter

    Description

    Role name

    The name of the role. Enter a custom name.

    Indices

    The indexes that you want to manage. Select logstash-*.

    Privileges

    The permissions that you want to grant to the role. Select read, write, create, delete, and create_index.

  5. Click Create role.

Create a user

In this example, a user for monitoring Logstash is created. You can use one of the following methods to create a user.

Important

By default, Elasticsearch disables the logstash_system user, and you are not allowed to create a user named logstash_system. You must create a user that is assigned the logstash_system role.

  • Method 1: Create a user in the Kibana console

    1. Log on to the Kibana console of the Elasticsearch cluster.

      For more information, see Log on to the Kibana console.

    2. Click the image icon in the upper-left corner of the page that appears, and choose Management > Stack Management.

    3. In the left-side navigation pane, choose Security > Users.

    4. On the Users page, click Create user in the upper-right corner.

    5. On the Create user page, configure the parameters.

      The following table describes some parameters that are displayed.

      Parameter

      Description

      Username

      The name of the user. Enter a custom name. Example: logstash_system_monitor.

      Roles

      The role that is assigned to the user. Select logstash_system.

    6. Click Create user.

  • Method 2: Create a user by using the command line interface (CLI)

    Connect to the ECS instance and run the following command to create a user:

    curl -u elastic:<es-password> -XPOST http://<instanceId>.elasticsearch.aliyuncs.com:9200/_xpack/security/user/logstash_system_monitor -d '{"password" : "<logstash-monitor-password>","roles" : ["logstash_system"],"full_name" : "<your full name>"}'

    Parameter

    Description

    es-password

    The password of the Elasticsearch cluster. This password is also used to log on to the Kibana console of the cluster.

    <instanceId>

    The ID of the Elasticsearch cluster. You can query the cluster ID on the Basic Information page of the Elasticsearch cluster.

    <logstash-monitor-password>

    The password of the logstash_system_monitor user.

    <your full name>

    The full name of the user.

    After the user is created, the result shown in the following figure is returned.添加用户成功

Step 2: Install Logstash on the ECS instance

  1. Connect to the ECS instance.

    For more information, see Connect to an ECS instance.

  2. Download the Logstash installation package.

    Visit the Elastic website and download the Logstash installation package that has the same version as the Elasticsearch cluster.

    wget https://artifacts.elastic.co/downloads/logstash/logstash-8.5.1-linux-x86_64.tar.gz
    Note

    Command for Logstash V7.4:

    wget https://artifacts.elastic.co/downloads/logstash/logstash-7.4.0.tar.gz
  3. Decompress the Logstash installation package.

    In this example, Logstash V8.5.1 is used.

    tar -xzvf logstash-8.5.1-linux-x86_64.tar.gz

Step 3: Configure and run a Logstash pipeline

  1. Go to the Logstash directory.

    cd logstash-8.5.1
  2. Create a .conf file.

    touch test.conf
  3. Configure the test.conf file.

    The following sample code provides a configuration example:

    input {
        file {
            path => "/your/file/path/xxx"
            }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => "<http://<instanceId>.elasticsearch.aliyuncs.com:9200>"
      	index => "<index-name>"
        user => "<user-name>"
        password => "<logstash-password>"
      }
    }

    Parameter

    Description

    path

    The path of the log file. In this example, /var/log/messages is used.

    hosts

    The endpoint of the Elasticsearch cluster. Replace instanceId with the ID of the Elasticsearch cluster. You can query the cluster ID on the Basic Information page of the cluster. Example: http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200.

    index

    The name of the index to which the data in the log file is written. You can specify a custom name.

    user

    The username that is used to access the Elasticsearch cluster. In this example, elastic is used.

    Important
    • To ensure system security, we recommend that you use a custom user instead of the elastic user in actual business.

    • The username must be enclosed in a pair of double quotation marks ("). This avoids potential errors caused by special characters in the username when you run the Logstash pipeline.

    password

    The password that is used to access the Elasticsearch cluster.

    Important

    The password must be enclosed in a pair of double quotation marks ("). This avoids potential errors caused by special characters in the password when you run the Logstash pipeline.

    Logstash provides a variety of input, filter, and output plug-ins. You need only to simply configure these plug-ins for data transfer. For more information, see Structure of a Config File.

  4. Run the Logstash pipeline.

    bin/logstash -f test.conf

    If the command is successfully run, Logstash automatically detects changes to the log file and synchronizes the changes to the Elasticsearch cluster. Logstash automatically synchronizes data to the Elasticsearch cluster whenever data is added to the monitored log file.

Step 4: View the migrated data

  1. Log on to the Kibana console of the Elasticsearch cluster.

    For more information, see Log on to the Kibana console.

  2. Click the image icon in the upper-left corner of the page that appears, and choose Management > Dev Tools.

  3. On the Console tab of the page that appears, run the following command:

    GET /logstash-*/_search

Step 5: (Optional) Monitor the Logstash node

  1. Navigate to the config folder in the Logstash directory.

    cd logstash-8.5.1/config
  2. Configure the logstash.yml file.

    1. Open the logstash.yml file.

      vim logstash.yml
    2. In the logstash.yml file, delete the annotations of the parameters that are listed in the following table. Then, configure the parameters.

      xpack.monitoring.enabled:true
      xpack.monitoring.elasticsearch.username:"<user-name>"
      xpack.monitoring.elasticsearch.password:"<logstash-password>"
      xpack.monitoring.elasticsearch.hosts:["http://es-cn-nwy3fwjp8xxxxxxx.elasticsearch.aliyuncs.com:9200"]

      Parameter

      Description

      xpack.monitoring.enable

      Set this parameter to true. Default value: false.

      xpack.monitoring.elasticsearch.username

      The name of the user that you created for Logstash monitoring. In this example, the elastic user is used.

      Important

      To ensure system security, we recommend that you use a custom user instead of the elastic user in actual business.

      xpack.monitoring.elasticsearch.password

      The password of the user that you created for Logstash monitoring.

      xpack.monitoring.elasticsearch.hosts

      The endpoint of the Elasticsearch cluster. Example: http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200.

  3. Go to the Logstash directory and launch the Logstash service.

    cd ../
    bin/logstash -f test.conf
  4. Log on to the Kibana console of the Elasticsearch cluster.

    For more information, see Log on to the Kibana console.

  5. Click the image icon in the upper-left corner of the page that appears, and choose Management > Stack Monitoring.

  6. (Optional) In the Create rules dialog box, click OK.

  7. On the Cluster overview page, view Logstash monitoring data.

References