Use a self-managed Logstash to migrate data from log files - Elasticsearch

When you need to migrate log file data to Alibaba Cloud Elasticsearch but don't require strict real-time guarantees, a self-managed Logstash instance on Elastic Compute Service (ECS) is a practical solution. This topic shows you how to deploy Logstash on ECS, configure a pipeline to read log files, and stream records into an Elasticsearch cluster.

How a Logstash pipeline works

A Logstash pipeline has three stages:

[Input] → [Filter] → [Output]

Input: reads data from a source, such as a log file
Filter: parses and transforms the data (optional)
Output: writes the processed data to a destination, such as Elasticsearch

All examples in this topic use a file input, an optional filter, and an Elasticsearch output.

Prerequisites

Before you begin, make sure you have:

An Alibaba Cloud Elasticsearch cluster (Standard Edition, V8.5.1 used in this topic) with Auto Indexing enabled. See Create an Alibaba Cloud Elasticsearch cluster and Access and configure an Elasticsearch cluster.
An ECS instance in the same region, zone, and virtual private cloud (VPC) as the Elasticsearch cluster, with network access to both services. See Create an instance by using the wizard.
JDK Temurin 17 or later installed on the ECS instance. See Install the JDK and Compatibility between Logstash and JVM.
Log data to migrate. This topic uses ECS log data as the example source.

Step 1: (Optional) Create a role and a user

Connecting a self-managed Logstash cluster to Alibaba Cloud Elasticsearch requires an Elasticsearch user. Use the built-in elastic user for testing, or create a dedicated user for production.

Important

Use a dedicated user in production. The elastic user has unrestricted superuser access, which poses a security risk.

Important

Elasticsearch disables the logstash_system user by default. Do not create a user named logstash_system. Instead, create a user and assign it the logstash_system role.

Create a role

Create a role with read and write permissions on logstash-* indexes.

Log on to the Kibana console of the Elasticsearch cluster. See Log on to the Kibana console.
Click the icon in the upper-left corner, and choose Management > Stack Management.
In the left-side navigation pane, choose Security > Roles.

Click Create role in the upper-right corner of the Roles page. Configure the following parameters:

Parameter	Description
Role name	A custom name for the role
Indices	`logstash-*`
Privileges	`read`, `write`, `create`, `delete`, `create_index`

Click Create role.

Create a user

Create a user for Logstash monitoring and assign it the logstash_system role. Use either the Kibana console or the CLI.

Option 1: Kibana console

Log on to the Kibana console. See Log on to the Kibana console.
Click the icon, and choose Management > Stack Management.
In the left-side navigation pane, choose Security > Users.
Click Create user in the upper-right corner of the Users page.

Configure the following parameters:

Parameter	Description
Username	A custom name for the user, such as `logstash_system_monitor`
Roles	`logstash_system`

Click Create user.

Option 2: CLI

Run the following command from the ECS instance to create the user. Replace the placeholders with your actual values.

curl -u elastic:<es-password> -XPOST http://<instanceId>.elasticsearch.aliyuncs.com:9200/_xpack/security/user/logstash_system_monitor -d '{"password" : "<logstash-monitor-password>","roles" : ["logstash_system"],"full_name" : "<your full name>"}'

Placeholder	Description
`<es-password>`	Password of the `elastic` user, also used to log on to Kibana
`<instanceId>`	ID of the Elasticsearch cluster, found on the Basic Information page
`<logstash-monitor-password>`	Password for the `logstash_system_monitor` user
`<your full name>`	Display name for the user

After the user is created, the result shown in the following figure is returned.

Step 2: Install Logstash on the ECS instance

Connect to the ECS instance. See Connect to an ECS instance.

Download the Logstash package that matches your Elasticsearch cluster version from the Elastic website. For Logstash V8.5.1:

wget https://artifacts.elastic.co/downloads/logstash/logstash-8.5.1-linux-x86_64.tar.gz

For Logstash V7.4.0:

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.4.0.tar.gz

Extract the package:

tar -xzvf logstash-8.5.1-linux-x86_64.tar.gz

Step 3: Configure and run a Logstash pipeline

Go to the Logstash directory:
```
cd logstash-8.5.1
```
Create the pipeline configuration file:
```
touch test.conf
```

Add the following configuration to test.conf:

Parameter	Description
`path`	Path to the log file to monitor. Example: `/var/log/messages`
`hosts`	Elasticsearch cluster endpoint. Replace `<instanceId>` with your cluster ID. Example: `http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200`
`index`	Target index name. Use `logstash-*` to match the role permissions created in Step 1
`user`	Elasticsearch username. Enclose the value in double quotation marks to avoid errors from special characters
`password`	Elasticsearch password. Enclose the value in double quotation marks to avoid errors from special characters

input {
  file {
    path => "/var/log/messages"
  }
}
filter {
}
output {
  elasticsearch {
    hosts => ["http://<instanceId>.elasticsearch.aliyuncs.com:9200"]
    index => "<index-name>"
    user => "<user-name>"
    password => "<logstash-password>"
  }
}

Logstash supports a wide range of input, filter, and output plugins. For the full configuration syntax, see Structure of a config file.

Start the pipeline:
```
bin/logstash -f test.conf
```
Logstash watches the configured log file and ships new lines to Elasticsearch as they are written.

Step 4: View the migrated data

Log on to the Kibana console. See Log on to the Kibana console.
Click the icon, and choose Management > Dev Tools.
On the Console tab, run:
```
GET /logstash-*/_search
```
Results confirm that log data has been indexed into Elasticsearch.

Step 5: (Optional) Monitor the Logstash node

Enable X-Pack monitoring to view Logstash metrics in the Kibana Stack Monitoring dashboard.

Open the monitoring configuration file:
```
vim logstash-8.5.1/config/logstash.yml
```

Uncomment and set the following parameters:

Parameter	Description
`xpack.monitoring.enabled`	Set to `true` to enable monitoring. Default: `false`
`xpack.monitoring.elasticsearch.username`	The name of the user that you created for Logstash monitoring. In this example, the elastic user is used
`xpack.monitoring.elasticsearch.password`	Password of the monitoring user
`xpack.monitoring.elasticsearch.hosts`	Elasticsearch cluster endpoint. Example: `http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200`

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: "<user-name>"
xpack.monitoring.elasticsearch.password: "<logstash-password>"
xpack.monitoring.elasticsearch.hosts: ["http://es-cn-nwy3fwjp8xxxxxxx.elasticsearch.aliyuncs.com:9200"]

Restart Logstash with the updated configuration:
```
cd logstash-8.5.1
bin/logstash -f test.conf
```
Log on to the Kibana console and click the icon, then choose Management > Stack Monitoring.
(Optional) In the Create rules dialog box, click OK.
On the Cluster overview page, view Logstash monitoring data.

What's next

References

Plugin configuration overview