Migrate Elasticsearch data with elasticsearch-dump - Elasticsearch

Use the open-source elasticsearch-dump tool to migrate index settings, mappings, and data between Alibaba Cloud Elasticsearch clusters or between on-premises and cloud environments.

Overview

elasticsearch-dump is a command-line tool that exports data from one Elasticsearch index into another index or a local file.

Best for:

Migrating small volumes of data.
Moving specific indexes.
Backing up mappings or settings to local JSON files.

Official documentation: elasticdump.

Prerequisites

Source/Destination clusters: Alibaba Cloud Elasticsearch clusters must be created. See Create an Alibaba Cloud Elasticsearch cluster.
Auto-indexing: The destination cluster must have Auto Indexing enabled, or the target index must be created manually in advance. Configure the YML file.
Migration node: An Elastic Compute Service (ECS) is required to run the tool.
- If the ECS instance and Elasticsearch cluster are in the same Virtual Private Cloud (VPC), use internal endpoints for faster, free data transfer.
- If they are in different regions/VPCs, use public endpoints and ensure the ECS IP is added to the cluster whitelist. See Manage IP address whitelists.

Install elasticsearch-dump

Connect to the ECS instance.

Connect to a Linux instance by using a password or key.

Install Node.js.

Download the package:

wget https://nodejs.org/dist/v16.18.0/node-v16.18.0-linux-x64.tar.xz

Decompress and move:
```
tar -xf node-v16.18.0-linux-x64.tar.xz
```

Configure environment variables:

Temporary (current session only):

export PATH=$PATH:/root/node-v16.18.0-linux-x64/bin/

Permanent (add to shell profile):

vim ~/.bash_profile
export PATH=$PATH:/root/node-v16.18.0-linux-x64/bin/
source ~/.bash_profile

Install elasticsearch-dump:
```
npm install elasticdump -g
```

Examples

Important

If your password contains special characters (e.g., #, $, @), standard URL strings may fail. See the FAQ and troubleshooting section for the --httpAuthFile solution.

Migrate data between clusters (cloud-to-cloud)

To fully migrate an index, run the command for settings, mappings, and data in that order.

Migrate index settings:

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=settings

Migrate mappings:

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=mapping

Migrate documents (data)

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=data

Export to local file (backup)

Migrate settings

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=settings

Migrate mappings

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=mapping

Migrate documents (data)

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=data

Migrate data based on a query

elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> ----searchBody="<YourQuery>"

Import from local file (restore)

Restore documents

elasticdump --input=<YourLocalFile> --output=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --type=data

Parameter reference

Parameter	Description
`--input` / `--output`	Source and destination. Can be a cluster URL or a local file path. Important When exporting to a local file, elasticsearch-dump creates the file at the specified path. Ensure the destination filename is unique in the target directory.
<YourEsHost>/<OtherEsHost>	The endpoint (internal or public) of your Elasticsearch cluster (e.g., `es-cn-abc.public.elasticsearch.aliyuncs.com:9200`). View the basic information of a cluster.
<UserName>/<OtherName>	Cluster username (default is `elastic`).
<YourPassword>/<OtherPassword>	Cluster password.
`--type`	The type of migration: `settings`, `mapping`, or `data`.
`--searchBody`	Filter data using a Query DSL. Example: `{"query":{"term":{"user":"admin"}}}`

FAQ and troubleshooting

Q: Error: getaddrinfo ENOTFOUND elastic

[root@i-xxx ~]# elasticdump --input=http://"xxx.public.elasticsearch.aliyuncs.com:9200/customers --output=/root/customers.json --type=settings
Tue, 04 Jul 2023 08:42:53 GMT | starting dump
(node:25448) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
Tue, 04 Jul 2023 08:43:13 GMT | Error Emitted => getaddrinfo ENOTFOUND elastic
Tue, 04 Jul 2023 08:43:13 GMT | Error Emitted => getaddrinfo ENOTFOUND elastic
Tue, 04 Jul 2023 08:43:13 GMT | Total Writes: 0
Tue, 04 Jul 2023 08:43:13 GMT | dump ended with error (get phase) => Error: getaddrinfo ENOTFOUND elastic

Cause: This usually occurs when your password contains special characters (like #, !, or @) that break the URL structure.

Solutions: Use an authentication file

Create a file named auth.ini:
```
user=elastic
password="Your#Complex$Password"
```
Note
The password must be enclosed in double quotation marks (").

Run elasticdump using the --httpAuthFile flag:

elasticdump --input=http://es-*****.public.elasticsearch.aliyuncs.com:9200/customers --output=/root/customers.json --httpAuthFile=/root/auth.ini --type=settings

Q: URIError: URI malformed

Cause: elasticsearch-dump cannot parse usernames and passwords that contain special characters.

Solutions:

Method 1: Remove the special characters in the username or password.
Method 2: Log on to the Kibana console, create a user with the required permissions, and use it for the migration.