Use elasticsearch-dump to migrate data - Elasticsearch - Alibaba Cloud Documentation Center

You can use the elasticsearch-dump tool to migrate the settings, mappings, and documents of indexes between Alibaba Cloud Elasticsearch clusters or between an on-premises machine and an Alibaba Cloud Elasticsearch cluster. This topic describes how to install and use elasticsearch-dump.

Background information

elasticsearch-dump is an open source tool used to migrate Elasticsearch data. For more information, see the official documentation of elasticsearch-dump.

Prerequisites

An Alibaba Cloud Elasticsearch cluster is created. For more information, see Create an Alibaba Cloud Elasticsearch cluster.
The Auto Indexing feature is enabled for the destination Alibaba Cloud Elasticsearch cluster, or an index is created in the Alibaba Cloud Elasticsearch cluster. For more information about how to enable the Auto Indexing feature, see Configure the YML file.
An Elastic Compute Service (ECS) instance is created. The ECS instance is used to install elasticsearch-dump. For more information, see Create an instance by using the wizard.

Use scenarios

elasticsearch-dump is used to migrate small volumes of data in scenarios with a small number of indexes.

Install elasticsearch-dump

Connect to the ECS instance.
For more information, see Connect to a Linux instance by using a password or key.
Install Node.js.
1. Download the installation package.
```
wget https://nodejs.org/dist/v16.18.0/node-v16.18.0-linux-x64.tar.xz
```
2. Decompress the package.
```
tar -xf node-v16.18.0-linux-x64.tar.xz
```
3. Configure environment variables.
  - If you want the environment variables to temporarily take effect, run the following command:
```
export PATH=$PATH:/root/node-v16.18.0-linux-x64/bin/
```
  - If you want the environment variables to permanently take effect, run the following commands:
```
vim ~/.bash_profile
export PATH=$PATH:/root/node-v16.18.0-linux-x64/bin/
source ~/.bash_profile
```
Install elasticsearch-dump.
```
npm install elasticdump -g
```

Examples

Note

If the password that is used to access the source or destination Alibaba Cloud Elasticsearch cluster contains special characters such as # and $, the request is blocked by Alibaba Cloud Web Application Firewall (WAF), and an error is reported after the related command is run. For more information about the reported error and troubleshooting method, see FAQ.

Migrate data to an Alibaba Cloud Elasticsearch cluster

Migrate the settings of an index

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=settings

Migrate the mappings of an index

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=mapping

Migrate the documents of an index

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=data

Migrate data to your on-premises machine

Migrate the settings of an index

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=settings

Migrate the mappings of an index

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=mapping

Migrate the documents of an index

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=data

Migrate data based on a query

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> ----searchBody="<YourQuery>"

Restore data from your on-premises machine to an Alibaba Cloud Elasticsearch cluster

Restore the documents of an index

elasticdump --input=<YourLocalFile> --output=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --type=data

For more information about other data migration scenarios, see the official documentation of elasticsearch-dump.

The following table describes the parameters used in the preceding examples.

Parameter	Description
<YourEsHost>/<OtherEsHost>	The internal or public endpoint of the source or destination Alibaba Cloud Elasticsearch cluster. You can obtain the internal or public endpoint of an Alibaba Cloud Elasticsearch cluster from the Basic Information page of the cluster. For more information, see View the basic information of a cluster. Note If the server on which elasticsearch-dump is installed resides in the same virtual private cloud (VPC) as the source and destination Elasticsearch clusters, you can use the internal endpoint. Otherwise, you must use the public endpoint and configure a public IP address whitelist. For more information, see Configure a public or private IP address whitelist for an Elasticsearch cluster.
<UserName>/<OtherName>	The username that is used to access the source or destination Alibaba Cloud Elasticsearch cluster. The default username for an Alibaba Cloud Elasticsearch cluster is elastic.
<YourPassword>/<OtherPassword>	The password that is used to access the source or destination Alibaba Cloud Elasticsearch cluster. The password is specified when you create the cluster. If you forget the password, you can reset it. For more information about the procedure and precautions for resetting a password, see Reset the access password for an Elasticsearch cluster.
<YourEsIndex>/<OtherEsIndex>	The name of the source or destination index.
<YourLocalFile>	The on-premises path that is used to store the data to be migrated. Example: `/data/test.json`. Important When you migrate data to your on-premises machine, elasticsearch-dump automatically generates the destination file in the specified path. Therefore, before you migrate data to the on-premises machine, you must make sure that the name of the destination file is unique in the related directory.
<YourQuery>	The query statement. Example: `{\"query\":{\"term\":{\"_id\":\"100\"}}}`.

FAQ

Q: What do I do if the error Error: getaddrinfo ENOTFOUND elastic is reported?

A: The HTTP authentication password contains special characters. As a result, a URL parsing error occurs. To resolve this issue, you need to add an HTTP authentication file.

Enter the username and password for accessing the Elasticsearch cluster in the authentication file auth.ini. Example:
```
user=elastic
password="Es#123456"
```
Note
The password must be enclosed in a pair of double quotation marks (").

Run the following command and specify the authentication file by using the --httpAuthFile parameter:

elasticdump --input=http://es-*****.public.elasticsearch.aliyuncs.com:9200/customers --output=/root/customers.json --httpAuthFile=/root/auth.ini --type=settings

Q: What do I do if the error URIError: URI malformed is reported?

A: The elasticsearch-dump plug-in cannot parse usernames and passwords that contain special characters. If the username or password of your Alibaba Cloud Elasticsearch cluster contains special characters, the system reports the preceding error. You can use one of the following methods to troubleshoot the issue:

Method 1: Remove the special characters in the username or password to change the username or password.
Method 2: Log on to the Kibana console of the cluster, create a user, grant the required permissions to the user, and then use the new user for data migration.