Create automatic snapshots and restore data from automatic snapshots - Elasticsearch

Alibaba Cloud Elasticsearch provides the Auto Snapshot feature. This feature allows you to specify the creation interval and time of snapshots. After you specify them, the system creates snapshots for your Elasticsearch cluster based on the specified interval and time to ensure data security. These snapshots are called automatic snapshots. You can restore data from the snapshots to the cluster. This topic describes how to use the Auto Snapshot feature, view the information of automatic snapshots, and restore data from automatic snapshots.

Background information

The data backup and restoration of Alibaba Cloud Elasticsearch clusters depend on the elasticsearch-repository-oss plug-in. By default, the plug-in is installed on Alibaba Cloud Elasticsearch clusters and cannot be removed. For more information about this plug-in, see elasticsearch-repository-oss.

Precautions

The first snapshot for a cluster is a full copy of the data in the cluster. Subsequent snapshots store only incremental data. Therefore, more time is required to create the first snapshot than a subsequent snapshot. The specific time depends on the data volume.
Snapshots store only index data. The following information of an Elasticsearch cluster is not stored in snapshots: monitoring data (such as indexes whose names start with .monitoring or .security_audit), metadata, translogs, configurations, software packages, built-in and custom plug-ins, and logs.
The system stores only the automatic snapshots that are created during the last seven days.
You can restore data from automatic snapshots only to the Elasticsearch cluster for which the snapshots are created. For more information about how to restore data to a cluster other than the original cluster, see Create manual snapshots and restore data from manual snapshots or Configure a shared OSS repository.

Enable the Auto Snapshot feature

Log on to the Elasticsearch console.
In the left-side navigation pane, click Elasticsearch Clusters.
Navigate to the desired cluster.
1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
2. In the left-side navigation pane, click Elasticsearch Clusters. On the Elasticsearch Clusters page, find the cluster and click its ID.
In the left-side navigation pane of the page that appears, click Snapshots.
In the Snapshots (Free Trial) section, turn on Auto Snapshot.
Click Set on the right side of the Auto Snapshot Period parameter.

In the Auto Snapshot Configuration panel, configure the Frequency parameter.

Specify an interval for automatic snapshot creation


Frequency	Description
Every 30 Minutes	The system creates snapshots at 30-minute intervals.
Daily	The system creates snapshots every day. You can customize creation time.
Custom	The system creates snapshots based on the interval and time you specify.

Important The system uses the system time of the region where your cluster resides to create snapshots.

Click OK.

View the information of automatic snapshots

After you enable the Auto Snapshot feature, you can log on to the Kibana console of your cluster and call the _snapshot API to view the information of the automatic snapshots of the cluster.

Run the following command to query the repository that stores the automatic snapshots of your cluster:

GET _snapshot

If the command is successfully run, the following result is returned:

{
  "aliyun_auto_snapshot" : {
    "type" : "oss",
    "settings" : {
      "compress" : "true",
      "base_path" : "cf95b9-185320276651****/es-cn-09k2053us0003****",
      "endpoint" : "http://oss-cn-hangzhou-internal.aliyuncs.com"
    }
  }
}


Parameter	Description
aliyun_auto_snapshot	The name of the repository. The repository is created when the first snapshot is created. The name of the repository that is used to store the automatic snapshots of Alibaba Cloud Elasticsearch clusters is fixed as aliyun_auto_snapshot. Note A repository can store multiple snapshots, each of which can contain the data of all indexes, some indexes, or a single index.
type	The storage of snapshots. The value oss indicates that snapshots are stored in Object Storage Service (OSS).
compress	Specifies whether the data compression feature is enabled for snapshots. Valid values: true: indicates that the data compression feature is enabled for snapshots. This feature applies only to index metadata, including the mappings and settings of indexes. false: indicates that the data compression feature is disabled for snapshots. The default value is false.
base_path	The location of snapshots in OSS.
endpoint	The endpoint of the OSS bucket that stores snapshots.

Run the following command to query the information of all the snapshots stored in the aliyun_auto_snapshot repository:

GET _snapshot/aliyun_auto_snapshot/_all

If the command is successfully run, the following result is returned:

{
  "snapshots": [
    {
      "snapshot": "es-cn-09k2053us0003****_20210117030003",
      "uuid": "vIdSCkthTeGa0nSj4D****",
      "version_id": 5050399,
      "version": "5.5.3",
      "indices": [
        ".kibana"
      ],
      "state": "SUCCESS",
      "start_time": "2018-06-28T01:22:39.609Z",
      "start_time_in_millis": 1530148959609,
      "end_time": "2018-06-28T01:22:39.923Z",
      "end_time_in_millis": 1530148959923,
      "duration_in_millis": 314,
      "failures": [],
      "shards": {
        "total": 1,
        "failed": 0,
        "successful": 1
      }
    },
    {
      "snapshot": "es-cn-09k2053us0003****_20210118030004",
      "uuid": "XKO_Uwz_Qu6mZrU3Am****",
      "version_id": 5050399,
      "version": "5.5.3",
      "indices": [
        ".kibana"
      ],
      "state": "SUCCESS",
      "start_time": "2018-06-28T01:25:00.764Z",
      "start_time_in_millis": 1530149100764,
      "end_time": "2018-06-28T01:25:01.482Z",
      "end_time_in_millis": 1530149101482,
      "duration_in_millis": 718,
      "failures": [],
      "shards": {
        "total": 1,
        "failed": 0,
        "successful": 1
      }
    }
  ]
}

Important The system uses the system time of the current region to create snapshots. However, the time in the returned result is in UTC. A time zone difference exists between the system time and UTC. You can convert the time in the returned result based on the time zone difference. For example, the time zone difference between the system time of the China (Beijing) region and UTC is 8 hours. In this case, the system time of the China (Beijing) region is UTC+0800.

In the returned result, the state parameter indicates the status of a snapshot. A snapshot can be in one of the following states.


State	Description
IN_PROGRESS	The snapshot is being created.
SUCCESS	The snapshot is created, and all shards are stored.
FAILED	The snapshot fails to be created because some shards cannot be stored.
PARTIAL	The snapshot is created, but at least one shard fails to be stored.
INCOMPATIBLE	The snapshot version is incompatible with the cluster version.

Automatic snapshots also support the following default parameters that are not displayed.


Parameter	Description
max_snapshot_bytes_per_sec	The maximum number of bytes that can be backed up for a single node per second. Default value: 40mb.
max_restore_bytes_per_sec	The maximum number of bytes that can be restored to a single node per second. Default value: 40mb.
chunk_size	During snapshot creation, a large index is divided into multiple parts. This parameter specifies the maximum size of each part. Example values: 1g, 10m, and 5k. Default value: null. This value indicates that the size of each part is not limited.

Run the following command to query the information of a specific automatic snapshot stored in the aliyun_auto_snapshot repository:

GET _snapshot/aliyun_auto_snapshot/<snapshot>/_status

Replace <snapshot> with the name of the automatic snapshot, such as es-cn-09k2053us0003****_20210118030004. You can query the name of a snapshot by running the command that is used to query all snapshots.

If the command is successfully run, the following result is returned:

{
  "snapshots": [
    {
      "snapshot": "es-cn-09k2053us0003****_20210118030004",
      "repository": "aliyun_auto_snapshot",
      "uuid": "XKO_Uwz_Qu6mZrU3Am****",
      "state": "SUCCESS",
      "shards_stats": {
        "initializing": 0,
        "started": 0,
        "finalizing": 0,
        "done": 1,
        "failed": 0,
        "total": 1
      },
      "stats": {
        "number_of_files": 4,
        "processed_files": 4,
        "total_size_in_bytes": 3296,
        "processed_size_in_bytes": 3296,
        "start_time_in_millis": 1530148959688,
        "time_in_millis": 77
      },
      "indices": {
        ".kibana": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "number_of_files": 4,
            "processed_files": 4,
            "total_size_in_bytes": 3296,
            "processed_size_in_bytes": 3296,
            "start_time_in_millis": 1530148959688,
            "time_in_millis": 77
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "number_of_files": 4,
                "processed_files": 4,
                "total_size_in_bytes": 3296,
                "processed_size_in_bytes": 3296,
                "start_time_in_millis": 1530148959688,
                "time_in_millis": 77
              }
            }
          }
        }
      }
    }
  ]
}

Restore data from automatic snapshots

After automatic snapshots are created, you can call the _restore API to restore data from the snapshots to the cluster for which the snapshots are created.

Important Indexes whose names start with a period (.) are system indexes. We recommend that you do not restore these indexes. If you restore these indexes, you may fail to access the Kibana console.

In the Kibana console, run one of the following commands to restore indexes from a specific snapshot:

Restore all indexes from a specific snapshot that is stored in the aliyun_auto_snapshot repository. The restoration is performed in the background.
```
POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore
```
Replace <snapshot> with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.
Restore all indexes from a specific snapshot that is stored in the aliyun_auto_snapshot repository, and wait until the restoration is complete.
The _restore API asynchronously runs restoration tasks. After the system confirms that data restoration can be performed, it immediately returns a response and performs restoration in the background. You can configure the wait_for_completion parameter in the command to enable the system to return a response only after the restoration is complete.
```
POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore?wait_for_completion=true
```
Replace <snapshot> with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.

Restore a specific index from a specific snapshot that is stored in the aliyun_auto_snapshot repository, and rename the restored index. The restoration is performed in the background.

POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore
{
"indices": "index_1",
"rename_pattern": "index_(.+)",
"rename_replacement": "restored_index_$1"
}


Parameter	Description
<snapshot>	Replace it with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.
indices	The name of the index that you want to restore.
rename_pattern	Optional. This parameter specifies the regular expression that is used to match the name of the index you want to restore.
rename_replacement	Optional. This parameter specifies the regular expression that is used to rename a matching index.