Create Elasticsearch manual snapshots and restore data - Elasticsearch

Back up index data to OSS and restore on demand with Elasticsearch manual snapshots. Use them for data migration, point-in-time recovery, dev/test environment setup, or pre-operation backups.

Backup and restoration require the elasticsearch-repository-oss plugin, preinstalled on all Alibaba Cloud Elasticsearch instances and cannot be uninstalled.

Snapshots store only index data, excluding monitoring indexes (.monitoring, .security_audit), metadata, translog, configurations, packages, plugins, and logs. Run all commands in Kibana Dev Tools. Log on to the Kibana console.

Create a snapshot repository

A snapshot repository stores snapshots in an OSS bucket. Prepare a Standard-class OSS bucket in the same region as your Elasticsearch instance (Create a bucket). The RAM user requires the AliyunOSSFullAccess policy (Grant permissions to a RAM user).

Example: create a repository named my_backup.

Alibaba Cloud clusters

PUT _snapshot/my_backup/
{
    "type": "oss",
    "settings": {
        "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "xxxxxx",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

Self-managed 8.x clusters

Self-managed clusters require manual installation of the elasticsearch-repository-oss plugin. Prefix all parameter names with oss.client..

PUT /_snapshot/my_backup
{
    "type": "oss",
    "settings": {
        "oss.client.endpoint": "oss-cn-shanghai.aliyuncs.com",
        "oss.client.access_key_id": "xxx",
        "oss.client.secret_access_key": "xxx",
        "oss.client.bucket": "xxxxxx",
        "oss.client.base_path":"snapshot/",
        "oss.client.compress": true
    }
}

Parameters

Parameter	Description
endpoint	The internal endpoint of the OSS bucket. Regions and endpoints.
access_key_id	The AccessKey ID of the RAM user. Obtain an AccessKey pair.
secret_access_key	The AccessKey secret of the RAM user. Obtain an AccessKey pair.
bucket	The name of an existing OSS bucket.
compress	Compresses snapshot metadata (index mappings and settings). Does not affect data files. Default: `false`.
chunk_size	Maximum chunk size for uploads to OSS. Files exceeding this limit are split into multiple chunks.
base_path	The storage path within the bucket. Defaults to the root. Use subdirectories to isolate snapshots by cluster or environment, such as `snapshot/prod/` or `snapshot/dev/`.

Verify repository connectivity

POST _snapshot/my_backup/_verify

A successful response lists all nodes connected to the repository. On failure, verify the endpoint, bucket name, and RAM user permissions.

Get repository information

# Get information about all repositories
GET _snapshot
# Get information about a specific repository
GET _snapshot/my_backup

Create a snapshot

Snapshot of all indexes

PUT _snapshot/my_backup/snapshot_1

This creates a snapshot named snapshot_1 for all open indexes. The snapshot runs asynchronously. To wait for completion, add wait_for_completion=true:

PUT _snapshot/my_backup/snapshot_1?wait_for_completion=true

A repository can hold multiple snapshots. The first is a full backup; subsequent snapshots are incremental, storing only changed data.

Snapshot of specific indexes

PUT _snapshot/my_backup/snapshot_2
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
}

Parameter	Description
indices	Comma-separated list of indexes to back up. Wildcards supported, such as `logs-*`.
ignore_unavailable	If `true`, skips nonexistent indexes instead of failing.
include_global_state	If `false`, excludes the cluster's global state. Recommended for data-only backups.

Snapshot information

# View all snapshots
GET _snapshot/my_backup/_all

# View a specific snapshot
GET _snapshot/my_backup/snapshot_1

# View the detailed status of a snapshot, including statistics for each index and shard
GET _snapshot/my_backup/snapshot_1/_status

Delete a snapshot

DELETE _snapshot/my_backup/snapshot_1

If a snapshot is in progress, this command stops the process and deletes any partial data created.

Important

Do not delete snapshot files directly from the OSS console or other tools. This breaks the incremental backup chain and can corrupt all subsequent snapshots, making them unrecoverable.

Restore from a snapshot

Before restoring data:

Avoid restoring system indexes (prefixed with .), as this may break Kibana access.
Close or delete any existing index with the same name in the target cluster before restoring. Otherwise, restoration fails.
For cross-region restoration, first migrate the snapshot data in OSS to the target region (Migration implementation), then restore to the target cluster.

Create repository in target cluster

Create a repository in the target cluster pointing to the OSS backup location. Use the same parameters as in Create a snapshot repository.

PUT _snapshot/my_backup_restore/
{
    "type": "oss",
    "settings": {
        "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "xxxxxx",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

Restore a specific index

Restore and rename an index to avoid name conflicts:

POST /_snapshot/my_backup_restore/snapshot_1/_restore
{
 "indices": "index_1",
 "rename_pattern": "index_(.+)",
 "rename_replacement": "restored_index_$1"
}

Parameter	Description
indices	The index to restore from the snapshot. All other indexes are ignored.
rename_pattern	Regex pattern matching the index names to restore.
rename_replacement	Replacement pattern for renamed indexes. Supports capture group references.

Restore non-system indexes

POST _snapshot/my_backup_restore/snapshot_1/_restore
{"indices": "*,-.monitoring*,-.security*,-.kibana*,-.internal.alerts*,-.alerts*","ignore_unavailable": true}

This exclusion pattern covers common system indexes. Other versions may include additional ones, such as .ds-ilm-history-* and .slo-*. Adjust per your cluster. For 8.x, also exclude data stream backing indexes by adding -.ds*.

Restore all indexes

POST _snapshot/my_backup_restore/snapshot_1/_restore

The _restore API runs asynchronously. To wait for completion, add wait_for_completion=true:

POST _snapshot/my_backup_restore/snapshot_1/_restore?wait_for_completion=true

Restore to Indexing Service

When restoring to an Indexing Service instance, use ignore_index_settings to skip incompatible settings:

POST /_snapshot/my_backup_restore/snapshot_1/_restore
{
  "indices": "index_1",
  "ignore_index_settings": [
    "index.apack.cube.following_index"
  ]
}

Snapshot restoration status

Monitor restoration progress with the _recovery API.

# Check the restoration status of a specific index
GET restored_index_1/_recovery

# Check restoration status for all indexes (may include unrelated shards)
GET /_recovery/

Key output fields:

Field	Description
type	Recovery type. `snapshot` indicates recovery from a snapshot.
source	Source repository and snapshot.
percent	Restoration progress percentage.

Cancel restoration

Cancel a restoration by deleting the target index:

DELETE /restored_index_3