How to quickly use an Alibaba Cloud Logstash instance - Elasticsearch

Background information

Before you begin, make sure that you understand the following concepts:

Prerequisites

You must have an Alibaba Cloud account.

If you do not have an account, sign up for one.
You have a Virtual Private Cloud (VPC) and a VSwitch.

For more information, see Create an IPv4-based VPC.

Limitations

The source Elasticsearch cluster, the Logstash cluster, and the destination Elasticsearch cluster must reside in the same Virtual Private Cloud (VPC). If they are in different VPCs, you must configure a NAT gateway to enable connectivity over the internet. For more information, see Configure a NAT gateway for public data transfer.
The versions of the source Elasticsearch cluster, Logstash cluster, and destination Elasticsearch cluster must be compatible. For more information, see Product compatibility.

Procedure

Step 1: Prepare the environment

Create source and destination Elasticsearch clusters, enable auto-indexing on the destination cluster, and prepare test data.
Step 2: Create an Alibaba Cloud Logstash cluster

After you create an Alibaba Cloud Logstash cluster and its state becomes Active, you can create and run a pipeline.
Step 3: Create and run a pipeline

Create and configure an Alibaba Cloud Logstash pipeline and run it to synchronize data.
Step 4: View the synchronization result

View the data synchronization result in the Kibana console of the destination Elasticsearch cluster.

Step 1: Prepare the environment

Create Alibaba Cloud Elasticsearch clusters.
1. Log on to the Alibaba Cloud Elasticsearch console.
2. In the left-side navigation pane, click Elasticsearch Clusters.
3. On the Elasticsearch Clusters page, create two Alibaba Cloud Elasticsearch clusters.
  The two clusters serve as the source and destination for a Logstash pipeline. For more information, see Create an Alibaba Cloud Elasticsearch cluster. This topic uses a data migration solution of Alibaba Cloud Elasticsearch 6.7.0 > Alibaba Cloud Logstash 6.7.0 > Alibaba Cloud Elasticsearch 6.7.0. The provided scripts apply only to this solution. Compatibility is not guaranteed for other solutions. The following table describes the configurations of the created Alibaba Cloud Elasticsearch clusters.
  Note
  
  If you use a different solution, see Product compatibility to check for compatibility issues. If issues exist, upgrade your cluster version or purchase a new cluster.
  
  The default account for accessing an Alibaba Cloud Elasticsearch cluster is elastic, which is used as an example in this topic. If you use a custom user, you must grant that user the required roles and permissions. For more information, see Control user permissions with Elasticsearch X-Pack roles.
Enable auto-indexing on the destination Alibaba Cloud Elasticsearch cluster.

For more information, see Configure YML parameters.

Note
To ensure data security, Alibaba Cloud Elasticsearch sets the Auto Indexing feature to Not allowed by default. When Alibaba Cloud Logstash sends data, it creates an index by submitting the data, not by using the create index API. Therefore, before you use Alibaba Cloud Logstash to upload data, you must either set the Auto Indexing feature for your cluster to Permitted or create an index and mappings in advance.

Prepare test data.

Go to the Kibana console of the source Alibaba Cloud Elasticsearch cluster. On the Dev Tools page, on the Console tab, run the following commands to create the index and the test documents.

Important

For more information about how to log on to the Kibana console, see Log on to the Kibana console.
The following scripts use Elasticsearch 6.7 as an example and are for testing only. For sample scripts for Elasticsearch 7.0 or later, see Quick start for Elasticsearch.

Create an index named my_index with a type named my_type.

PUT /my_index
{
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
        "my_type" : {
            "properties" : {
              "post_date": {          
                   "type": "date"       
               },
              "tags": {
                   "type": "keyword"
               },
                "title" : {
                    "type" : "text"
                }
            }
        }
    }
}

Insert a document with the ID 1 into my_index.

PUT /my_index/my_type/1?pretty
{
  "title": "One", 
  "tags": ["ruby"],
  "post_date":"2009-11-15T13:00:00"
}

Insert a document with the ID 2 into my_index.

PUT /my_index/my_type/2?pretty
{
  "title": "Two", 
  "tags": ["ruby"],
  "post_date":"2009-11-15T14:00:00"
}

Step 2: Create an Alibaba Cloud Logstash cluster

Go to the Logstash Clusters page.
1. Go to the Logstash Clusters page.
2. In the top navigation bar, select the same region as the destination Alibaba Cloud Elasticsearch cluster.
3. In the left-side navigation pane, click Logstash Clusters.
On the Logstash Clusters page, click Create.
On the buy page configure the new cluster.
In this topic, the billing method is set to pay-as-you-go and the version is set to 6.7. Use the default values for the other parameters. For more information, see Create an Alibaba Cloud Logstash cluster.
Note
- During the early stages of development and functional testing, we recommend that you purchase a Pay-as-you-go cluster for testing.
- You can receive discounts when you purchase a Subscription cluster.
Read and select the terms of service, and click Buy Now.
After a success message appears, click Console.
In the top navigation bar, select the region where the cluster resides. In the left-side navigation pane, click Logstash Clusters. On the Logstash Clusters page, view the created cluster.

Step 3: Create and run a pipeline

After the state of the created Logstash cluster changes to Normal, you can create and run a pipeline to synchronize data.

On the Logstash Clusters page, find the destination cluster and click Pipelines in the Actions column.
In the Pipelines section, click Create Pipeline.

Enter a Pipeline ID and specify the Config Settings.

This topic uses the following pipeline configuration.

input {
    elasticsearch {
        hosts => ["http://es-cn-0pp1f1y5g000h****.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "*,-.monitoring*,-.security*,-.kibana*"
        docinfo => true
    }
}
filter {}
output {
    elasticsearch {
        hosts => ["http://es-cn-mp91cbxsm000c****.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "%{[@metadata][_index]}"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"
    }
    file_extend {
        path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
    }
}

Parameter	Description
hosts	The endpoint of the Alibaba Cloud Elasticsearch cluster. For the input block, use the format `http://<source_cluster_ID>.elasticsearch.aliyuncs.com:9200`. For the output block, use the format `http://<destination_cluster_ID>.elasticsearch.aliyuncs.com:9200`.
user	The username used to access the Alibaba Cloud Elasticsearch cluster. The default value is elastic.
password	The password of the specified user. You configure the password for the elastic user when you create the cluster. If you forget the password, you can reset it. For more information, see Reset the access password of a cluster.
index	The name of the index to synchronize. If you set this parameter to `,-.monitoring,-.security,-.kibana`, the pipeline synchronizes all indexes except system indexes whose names start with a period (`.`). The value `%{[@metadata][_index]}` indicates that the pipeline matches the index in the metadata. In this case, the name of the destination index is the same as the name of the source index. Note System indexes typically store monitoring logs for an Elasticsearch cluster and do not need to be synchronized.
docinfo	If you set this parameter to true, the pipeline extracts document metadata, such as the index, type, and ID.
document_type	The type of the destination index. The value `%{[@metadata][_type]}` indicates that the pipeline matches the type in the metadata. In this case, the type of the destination index is the same as the type of the source index.
document_id	The ID of the destination document. The value `%{[@metadata][_id]}` indicates that the pipeline matches the ID in the metadata. In this case, the ID of the destination document is the same as the ID of the source document.
file_extend	Optional. This parameter enables the debug logging feature. Use the path parameter to configure the output path for debug logs. We recommend that you configure this parameter. After it is configured, you can view the output directly in the console. If you do not configure this parameter, you must verify the output at the destination and then return to the console to modify the pipeline, which is inefficient. For more information, see Use the debugging feature for Logstash pipeline configurations. Important The file_extend parameter requires the logstash-output-file_extend plugin. Install or uninstall plugins. The path parameter defaults to a system-specified path. Do not modify it. You can also retrieve the path by clicking Start Configuration Debug.

Configuration file structure and supported data types may vary by version. Structure of a Config File.

Click Next step to configure pipeline parameters.

Set Pipeline Workers to the number of CPU cores in the cluster. Leave other parameters at their default values. For more information about the parameters, see Manage pipelines by using configuration files.
Click Save or Save and Deploy.
- Save: Saves the pipeline configuration but does not apply it. After saving, you are returned to the Pipelines page. In the Pipelines section, you can click Deploy Now in the Actions column to restart the instance and apply the configuration.
- Save and Deploy: Saves and deploys the configuration, restarting the instance to apply the changes.
In the success message that appears, click OK.

You can then view the created pipeline in the Pipelines list. Wait for the cluster update to complete and the pipeline Status to become Running. This indicates that Alibaba Cloud Logstash has started the synchronization.

Step 4: View the synchronization result

After the pipeline is configured and running, you can view the synchronization result in the Kibana console of the destination Alibaba Cloud Elasticsearch cluster.

Log on to the Kibana console of your Elasticsearch cluster.
For instructions, see Log on to the Kibana console.
Note
Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.
In the left navigation menu, choose Management > Dev Tools.

In the Console, run the following command to view the data synchronization result.

GET /my_index/_search
{
  "query": {
    "match_all": {}
  }
}

The expected result is as follows:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my_index",
        "_type" : "doc",
        "_id" : "aKG3bHABnsveUjNW2iXy",
        "_score" : 1.0,
        "_source" : {
          "post_date" : "2009-11-15T14:00:00",
          "@version" : "1",
          "@timestamp" : "2020-02-22T11:45:52.992Z",
          "tags" : [
            "ruby"
          ],
          "title" : "Two"
        }
      },
      ...
    ]
  }
}

If the data on the source and destination clusters is identical, the synchronization was successful. You can also run the GET _cat/indices?v command and compare the sizes of the same index on the source and destination clusters to verify that the synchronization was successful.

Elasticsearch:Quick start