Use Logstash to rename a field - Elasticsearch - Alibaba Cloud Documentation Center

In some business scenarios, you may need to rename some fields in indexes. For example, an Alibaba Cloud Elasticsearch cluster stores one or more fields whose names contain special characters, such as at signs (@), and you want to use DataWorks to migrate data from the cluster to another Alibaba Cloud Elasticsearch cluster. However, DataWorks cannot migrate the data of fields whose names contain special characters. In this case, you must rename the fields before you migrate the data. The topic describes how to use Logstash to rename a field.

Background information

You can use one of the following methods to rename a field:

Use a filter plug-in provided by Logstash.
In this topic, this method is used to rename the field @ctxt_user_info in a source index of a cluster to ctxt_user_info in a destination index of the cluster.
Rename the field when you use the reindex API to migrate data.

Prerequisites

An Alibaba Cloud Elasticsearch cluster is created.
For more information, see Create an Alibaba Cloud Elasticsearch cluster. In this example, an Elasticsearch V7.10 cluster is used.
An Alibaba Cloud Logstash cluster is created in the virtual private cloud (VPC) where the Elasticsearch cluster resides.
For more information, see Create an Alibaba Cloud Logstash cluster.

Test data is prepared.

In this example, the following test data is used. In the test data, the name of the field @ctxt_user_info in a source index of the Elasticsearch cluster contains an at sign (@).

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "product_info",
        "_type" : "_doc",
        "_id" : "rpN7fn0BKQKHRO31rK6C",
        "_score" : 1.0,
        "_source" : {
          "@ctxt_user_info" : "test1"
        }
      },
      {
        "_index" : "product_info",
        "_type" : "_doc",
        "_id" : "r5N7fn0BKQKHRO31rK6C",
        "_score" : 1.0,
        "_source" : {
          "@ctxt_user_info" : "test2"
        }
      }
    ]
  }
}

Procedure

Step 1: (Optional) Create a destination index in the Elasticsearch cluster
Step 2: Create and configure a Logstash pipeline
Step 3: Verify the result

Step 1: (Optional) Create a destination index in the Elasticsearch cluster

If you enable the Auto Indexing feature for the Elasticsearch cluster, you can skip this step.

Note

We recommend that you disable the Auto Indexing feature because the indexes that are automatically created may not meet your business requirements.

Log on to the Kibana console of the Elasticsearch cluster.
For more information, see Log on to the Kibana console.
Note
In this example, an Elasticsearch V7.10 cluster is used. Different operations may be required to log on to the Kibana console of clusters of other versions.
Go to the homepage of the Kibana console as prompted and click Dev tools in the upper-right corner.

On the Console tab of the page that appears, run the following command to create a destination index named product_info2 in the Elasticsearch cluster:

PUT /product_info2
{
    "settings": {
        "number_of_shards": 5,
        "number_of_replicas": 1
    },
    "mappings": {
        "properties": {
            "ctxt_user_info": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            }
        }
    }
}

Step 2: Create and configure a Logstash pipeline

Go to the Logstash Clusters page of the Alibaba Cloud Elasticsearch console.
Navigate to the desired cluster.
1. In the top navigation bar, select the region where the cluster resides.
2. On the Logstash Clusters page, find the cluster and click its ID.
In the left-side navigation pane of the page that appears, click Pipelines.
On the Pipelines page, click Create Pipeline.

In the Create Task wizard, enter a pipeline ID and configure the pipeline.

In this example, the following configurations are used for the pipeline:

input {
    elasticsearch {
        hosts => ["http://es-cn-tl32gid**********.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "product_info"
        docinfo => true
    }
}
filter {
    mutate {
        rename => { "@ctxt_user_info" => "ctxt_user_info" }
    }
}
output {
    elasticsearch {
        hosts => ["http://es-cn-tl32gid**********.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "product_info2"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"
    }
}

In the preceding configurations, the filter.mutate.rename parameter is used to rename the field in the source index of the Elasticsearch cluster.

For more information about pipeline configurations, see Use configuration files to manage pipelines and Logstash configuration files.

Warning After you configure the parameters, you must save the settings and deploy the pipeline. This triggers a restart of the Logstash cluster. Before you can proceed, make sure that the restart does not affect your services.

Click Save or Save and Deploy.
- Save: After you click this button, the system stores the pipeline settings and triggers a cluster change. However, the settings do not take effect. After you click Save, the Pipelines page appears. On the Pipelines page, find the created pipeline and click Deploy in the Actions column. Then, the system restarts the Logstash cluster to make the settings take effect.
- Save and Deploy: After you click this button, the system restarts the Logstash cluster to make the settings take effect.

Step 3: Verify the result

Log on to the Kibana console of the Elasticsearch cluster.
For more information, see Log on to the Kibana console.
Go to the homepage of the Kibana console as prompted and click Dev tools in the upper-right corner.

On the Console tab of the page that appears, run the following command to query the destination index in the Elasticsearch cluster:

GET product_info2/_search

If the command is successfully run, the following result is returned:

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "product_info2",
        "_type" : "_doc",
        "_id" : "r5N7fn0BKQKHRO31rK6C",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-12-03T04:14:26.872Z",
          "@version" : "1",
          "ctxt_user_info" : "test1"
        }
      },
      {
        "_index" : "product_info2",
        "_type" : "_doc",
        "_id" : "rpN7fn0BKQKHRO31rK6C",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-12-03T04:14:26.871Z",
          "@version" : "1",
          "ctxt_user_info" : "test2"
        }
      }
    ]
  }
}

The result shows that the field name that matches @ctxt_user_info in the source index of the Elasticsearch cluster is ctxt_user_info.