How to manually back up and restore data - Elasticsearch

Use the Elasticsearch snapshot feature to back up index data from your cluster to Alibaba Cloud Object Storage Service (OSS) and restore it when needed. Manual snapshots are ideal for data migration, point-in-time recovery, setting up development and testing environments, and creating backups before high-risk operations. Compared to automatic snapshots, manual snapshots offer more flexibility in scope and timing but require you to manage their lifecycle.

Data backup and restoration depend on the elasticsearch-repository-oss plugin. This plugin is installed by default on Alibaba Cloud Elasticsearch instances and cannot be uninstalled. For more information, see elasticsearch-repository-oss.

Snapshots store only index data. They do not store monitoring data (indexes with the .monitoring or .security_audit prefix), metadata, translog, instance configurations, software packages, plugins, or logs. All commands in this topic can be run in the Kibana console's Dev Tools. For more information, see Log on to the Kibana console.

Create a snapshot repository

A snapshot repository is a logical container for storing snapshots. Before you create a repository, you must have an OSS bucket in the same region as your Elasticsearch instance. We recommend using the Standard storage class. For instructions, see Create a bucket. The RAM user must have the AliyunOSSFullAccess policy. For instructions, see Grant permissions to a RAM user.

The following example creates a repository named my_backup.

Alibaba Cloud clusters

PUT _snapshot/my_backup/
{
    "type": "oss",
    "settings": {
        "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "xxxxxx",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

Self-managed 8.x clusters

For a self-managed cluster, you must first install the elasticsearch-repository-oss plugin. For more information, see elasticsearch-repository-oss plugin. When configuring a self-managed cluster, you must prefix the parameter names with oss.client..

PUT /_snapshot/my_backup
{
    "type": "oss",
    "settings": {
        "oss.client.endpoint": "oss-cn-shanghai.aliyuncs.com",
        "oss.client.access_key_id": "xxx",
        "oss.client.secret_access_key": "xxx",
        "oss.client.bucket": "xxxxxx",
        "oss.client.base_path":"snapshot/",
        "oss.client.compress": true
    }
}

Parameters

Parameter	Description
endpoint	The internal endpoint of the OSS bucket. For more information, see Regions and endpoints.
access_key_id	The AccessKey ID of the RAM user. For instructions, see Obtain an AccessKey pair.
secret_access_key	The AccessKey secret of the RAM user. For instructions, see Obtain an AccessKey pair.
bucket	The name of an existing OSS bucket.
compress	Specifies whether to compress snapshot metadata files, such as index mappings and settings. Data files are not affected. The default is `false`.
chunk_size	The size limit for each data chunk. Data larger than this size is split into multiple chunks before being uploaded to OSS.
base_path	The base path for the repository within the bucket. The default is the bucket root. You can specify a subdirectory to isolate snapshots from different clusters or environments, such as `snapshot/prod/` or `snapshot/dev/`.

Verify repository connectivity

POST _snapshot/my_backup/_verify

A successful request returns a list of all nodes that successfully connected to the repository. If an error occurs, check your endpoint, bucket name, and RAM user permissions.

Get repository information

# Get information about all repositories
GET _snapshot
# Get information about a specific repository
GET _snapshot/my_backup

Create a snapshot

Snapshot of all indexes

PUT _snapshot/my_backup/snapshot_1

This command creates a snapshot named snapshot_1 for all open indexes. The command returns immediately, and the snapshot runs in the background. To block the call until the backup is complete, add the wait_for_completion=true parameter:

PUT _snapshot/my_backup/snapshot_1?wait_for_completion=true

A repository can contain multiple snapshots. The first snapshot is a full backup. Subsequent snapshots are incremental backups that store only the data that has changed since the last snapshot, which reduces backup time.

Snapshot of specific indexes

PUT _snapshot/my_backup/snapshot_2
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
}

Parameter	Description
indices	Specifies the indexes to back up. Separate multiple indexes with commas. Wildcards are supported, such as `logs-*`.
ignore_unavailable	If `true`, Elasticsearch ignores specified indexes that do not exist instead of failing the request.
include_global_state	If `false`, the snapshot excludes the cluster's global state. This setting is recommended when you only need to back up data.

Snapshot information

# View all snapshots
GET _snapshot/my_backup/_all

# View a specific snapshot
GET _snapshot/my_backup/snapshot_1

# View the detailed status of a snapshot, including statistics for each index and shard
GET _snapshot/my_backup/snapshot_1/_status

Delete a snapshot

DELETE _snapshot/my_backup/snapshot_1

If a snapshot is in progress, this command stops the process and deletes any partial data created.

Important

Do not delete snapshot files directly from the OSS console or by using other tools. This action breaks the incremental backup chain and can corrupt all subsequent snapshots, making them unrecoverable.

Restore from a snapshot

Before you restore data, note the following:

We recommend not restoring system indexes, which are prefixed with a period (.), as this may cause Kibana access to fail.
If an index with the same name already exists in the target cluster, you must first close or delete the existing index. Otherwise, the restoration fails.
To perform a cross-region restoration, you must first migrate the snapshot data in OSS to the target region, and then restore it to the target cluster. For information about migrating data between OSS buckets, see Migration implementation.

Create repository in target cluster

Before you can restore data, you must create a repository in the target cluster that points to the OSS location of the backup. For parameter configurations, see Create a snapshot repository.

PUT _snapshot/my_backup_restore/
{
    "type": "oss",
    "settings": {
        "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "xxxxxx",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

Restore a specific index

Restore a specific index and rename it to avoid conflicts with existing indexes:

POST /_snapshot/my_backup_restore/snapshot_1/_restore
{
 "indices": "index_1",
 "rename_pattern": "index_(.+)",
 "rename_replacement": "restored_index_$1"
}

Parameter	Description
indices	Specifies the index to restore, ignoring all other indexes in the snapshot.
rename_pattern	A regular expression pattern that matches the names of the indexes to be restored.
rename_replacement	The replacement pattern for the new index name. Supports regular expression capture group references.

Restore non-system indexes

POST _snapshot/my_backup_restore/snapshot_1/_restore
{"indices": "*,-.monitoring*,-.security*,-.kibana*,-.internal.alerts*,-.alerts*","ignore_unavailable": true}

The preceding exclusion pattern covers common system indexes. Different cluster versions may have other system indexes, such as .ds-ilm-history-* and .slo-*. Adjust the exclusion list based on the system indexes in your cluster. For 8.x clusters, you also typically need to exclude data stream backing indexes, for example, by adding -.ds* to the list.

Restore all indexes

POST _snapshot/my_backup_restore/snapshot_1/_restore

The _restore API returns immediately, and the restoration runs in the background. To block the call until the restoration is complete, add the wait_for_completion=true parameter:

POST _snapshot/my_backup_restore/snapshot_1/_restore?wait_for_completion=true

Restore to Indexing Service

When you restore a snapshot to an Indexing Service instance, use the ignore_index_settings parameter to ignore incompatible index settings:

POST /_snapshot/my_backup_restore/snapshot_1/_restore
{
  "indices": "index_1",
  "ignore_index_settings": [
    "index.apack.cube.following_index"
  ]
}

Snapshot restoration status

Monitor the restoration status and progress with the _recovery API.

# Check the restoration status of a specific index
GET restored_index_1/_recovery

# Check restoration status for all indexes (may include unrelated shards)
GET /_recovery/

Key fields in the output:

Field	Description
type	The recovery type. `snapshot` indicates recovery from a snapshot.
source	The source repository and snapshot.
percent	The restoration progress, as a percentage.

Cancel restoration

To cancel a restoration, delete the index being restored:

DELETE /restored_index_3