Elasticsearch:Use Logstash to rename a field

Prerequisites

• An Alibaba Cloud Elasticsearch cluster is created. The source index and destination index can be in the same Elasticsearch cluster or in different Elasticsearch clusters. In this example, the source index and destination index are in the same Elasticsearch V7.10 cluster. For more information, see Create an Alibaba Cloud Elasticsearch cluster.

• An Alibaba Cloud Logstash cluster is created. The Logstash cluster must be in the same virtual private cloud (VPC) as the Elasticsearch cluster. For more information, see Create an Alibaba Cloud Logstash cluster.

Prepare test data

1. Log on to the Kibana console of the Elasticsearch cluster.

   For more information, see Log on to the Kibana console.

2. In the upper-left corner, click the icon and choose Management > Dev Tools

3. On the Console tab of the page that appears, run the following command to create a source index named product_info:

   PUT /product_info
   {
       "settings": {
           "number_of_shards": 5,
           "number_of_replicas": 1
       },
       "mappings": {
           "properties": {
               "ctxt_user_info": {
                   "type": "text",
                   "fields": {
                       "keyword": {
                           "type": "keyword",
                           "ignore_above": 256
                       }
                   }
               }
           }
       }
   }

4. Run the following command to insert test data into the source index:

   POST /product_info/_doc/_bulk
   {"index":{}}
   {"@ctxt_user_info":"test1"}
   {"index":{}}
   {"@ctxt_user_info":"test1"}

5. Run the following command to query data in the source index:

   GET /product_info/_search

   The returned result shows that the name of the field @ctxt_user_info in the source index contains an at sign (@).

   {
     "took" : 16,
     "timed_out" : false,
     "_shards" : {
       "total" : 5,
       "successful" : 5,
       "skipped" : 0,
       "failed" : 0
     },
     "hits" : {
       "total" : {
         "value" : 2,
         "relation" : "eq"
       },
       "max_score" : 1.0,
       "hits" : [
         {
           "_index" : "product_info",
           "_type" : "_doc",
           "_id" : "3BDMUZMBh7mRlA4aS0Nq",
           "_score" : 1.0,
           "_routing" : "74873",
           "_source" : {
             "@ctxt_user_info" : "test1"
           }
         },
         {
           "_index" : "product_info",
           "_type" : "_doc",
           "_id" : "3RDMUZMBh7mRlA4aS0Nq",
           "_score" : 1.0,
           "_routing" : "74873",
           "_source" : {
             "@ctxt_user_info" : "test1"
           }
         }
       ]
     }
   }

Procedure

To use Logstash to rename a field in an Elasticsearch index, perform the following steps:

1. Create a destination index in the Elasticsearch cluster to store data in the source index.

2. Configure a Logstash pipeline. In the configurations of the pipeline, specify a filter plug-in to rename the field @ctxt_user_info in the source index to ctxt_user_info in the destination index.

3. Verify that the field name is ctxt_user_info in the destination index.

Step 1: (Optional) Create a destination index in the Elasticsearch cluster

If you enable the Auto Indexing feature for the Elasticsearch cluster, you can skip this step. The indexes that are automatically created may not meet your business requirements. Therefore, we recommend that you disable the Auto Indexing feature.

Go to the Dev Tools page in the Kibana console of the Elasticsearch cluster. Then, run the following command to create a destination index named product_info2:

PUT /product_info2
{
    "settings": {
        "number_of_shards": 5,
        "number_of_replicas": 1
    },
    "mappings": {
        "properties": {
            "ctxt_user_info": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            }
        }
    }
}

Step 2: Create and configure a Logstash pipeline

1. Go to the Logstash Clusters page of the Alibaba Cloud Elasticsearch console.

2. Navigate to the desired cluster.

   1. In the top navigation bar, select the region where the cluster resides.

   2. On the Logstash Clusters page, find the cluster and click its ID.

3. In the left-side navigation pane of the page that appears, click Pipelines.

4. On the Pipelines page, click Create Pipeline.

5. On the Create page, enter a pipeline ID and configure the pipeline.

   In this example, the following configurations are used for the pipeline:

   input {
       elasticsearch {
           hosts => ["http://es-cn-tl32gid**********.elasticsearch.aliyuncs.com:9200"]
           user => "elastic"
           password => "your_password"
           index => "product_info"
           docinfo => true
       }
   }
   filter {
       mutate {
           rename => { "@ctxt_user_info" => "ctxt_user_info" }
       }
   }
   output {
       elasticsearch {
           hosts => ["http://es-cn-tl32gid**********.elasticsearch.aliyuncs.com:9200"]
           user => "elastic"
           password => "your_password"
           index => "product_info2"
           document_type => "%{[@metadata][_type]}"
           document_id => "%{[@metadata][_id]}"
       }
   }
                           

   In the preceding configurations, the filter.mutate.rename parameter is used to rename the field in the source index.

   For more information about pipeline configurations, see Use configuration files to manage pipelines and Logstash configuration files.

   Warning

   After you configure the parameters, you must save the settings and deploy the pipeline. This triggers a restart of the Logstash cluster. Before you can proceed, make sure that the restart does not affect your business.

6. Click Next.

7. Configure pipeline parameters.

8. Click Save or Save and Deploy.

   • Save: After you click this button, the system stores the pipeline settings and triggers a cluster change. However, the settings do not take effect. After you click Save, the Pipelines page appears. On the Pipelines page, find the created pipeline and click Deploy Now in the Actions column. Then, the system restarts the Logstash cluster to make the settings take effect.

   • Save and Deploy: After you click this button, the system restarts the Logstash cluster to make the settings take effect.

Step 3: Verify the result

Go to the Dev Tools page in the Kibana console of the Elasticsearch cluster. Then, run the following command to query data in the destination index product_info2:

GET product_info2/_search

The following result is returned. The result shows that the field name that matches @ctxt_user_info in the source index is ctxt_user_info.

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "product_info2",
        "_type" : "_doc",
        "_id" : "r5N7fn0BKQKHRO31rK6C",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-12-03T04:14:26.872Z",
          "@version" : "1",
          "ctxt_user_info" : "test1"
        }
      },
      {
        "_index" : "product_info2",
        "_type" : "_doc",
        "_id" : "rpN7fn0BKQKHRO31rK6C",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-12-03T04:14:26.871Z",
          "@version" : "1",
          "ctxt_user_info" : "test2"
        }
      }
    ]
  }
}