All Products
Search
Document Center

Elasticsearch:Create automatic snapshots and restore data from automatic snapshots

Last Updated:Feb 28, 2024

Alibaba Cloud Elasticsearch provides the Auto Snapshot feature. After you enable this feature for your Elasticsearch cluster, the system creates snapshots for the cluster based on the backup interval and time that you specify. This ensures data security. These snapshots are called automatic snapshots. If you encounter issues such as unexpected data deletion or application logic errors, you can restore data at a specific point of time from such a snapshot to the cluster. This topic describes how to enable the Auto Snapshot feature, disable the Auto Snapshot feature, and restore data from automatic snapshots.

Background information

The data backup and restoration of Alibaba Cloud Elasticsearch clusters depend on the elasticsearch-repository-oss plug-in. The plug-in is installed on Alibaba Cloud Elasticsearch clusters by default and cannot be removed. For more information about this plug-in, see elasticsearch-repository-oss.

Precautions

  • The first snapshot for a cluster is a full copy of the data in the cluster. Subsequent snapshots store only incremental data. Therefore, more time is required to create the first snapshot than a subsequent snapshot. The specific time depends on the data volume.

  • Snapshots store only index data. The following information of an Elasticsearch cluster is not stored in snapshots: monitoring data (such as indexes whose names start with .monitoring or .security_audit), metadata, translogs, configurations, software packages, built-in and custom plug-ins, and logs.

  • The system stores only the automatic snapshots that are created during the last seven days.

  • You can restore data from automatic snapshots only to the Elasticsearch cluster for which the snapshots are created. For information about how to restore data to a cluster other than the original cluster, see Create manual snapshots and restore data from manual snapshots or Configure a shared OSS repository.

  • When you disable the Auto Snapshot feature for an Elasticsearch cluster, the system creates a snapshot for data in the cluster. When you update backup time for data in the cluster, the system does not create a snapshot for the data.

Enable the Auto Snapshot feature

  1. Log on to the Alibaba Cloud Elasticsearch console.
  2. In the left-side navigation pane, click Elasticsearch Clusters.
  3. Navigate to the desired cluster.
    1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
    2. On the Elasticsearch Clusters page, find the cluster and click its ID.
  4. In the left-side navigation pane of the page that appears, click Snapshots.

  5. In the Snapshots (Free Trial) section, turn on Auto Snapshot.

  6. Click Set on the right side of the Auto Snapshot Period parameter.

  7. In the Auto Snapshot Configuration panel, configure the Frequency parameter.

    选择自动备份触发的时间

    Value of the Frequency parameter

    Description

    Every 30 Minutes

    The system creates snapshots at 30-minute intervals.

    Daily

    The system creates snapshots every day. You can customize creation time.

    Custom

    The system creates snapshots based on the interval and time you specify.

    Important

    The system uses the system time of the region where your cluster resides to create snapshots.

  8. Click OK.

View the information about automatic snapshots

After you enable the Auto Snapshot feature, you can log on to the Kibana console of your cluster and call the _snapshot API to view the information about the automatic snapshots of the cluster.

  • Run the following command to query the repository that stores the automatic snapshots of your cluster:

    GET _snapshot

    If the command is successfully run, the following result is returned:

    {
      "aliyun_auto_snapshot" : {
        "type" : "oss",
        "settings" : {
          "compress" : "true",
          "base_path" : "cf95b9-185320276651****/es-cn-09k2053us0003****",
          "endpoint" : "http://oss-cn-hangzhou-internal.aliyuncs.com"
        }
      }
    }
                                

    Parameter

    Description

    aliyun_auto_snapshot

    The name of the repository. The repository is created when the first snapshot is created. The name of the repository that is used to store the automatic snapshots of Alibaba Cloud Elasticsearch clusters is fixed as aliyun_auto_snapshot.

    Note

    A repository can store multiple snapshots, each of which can contain the data of all indexes, some indexes, or a single index.

    type

    The storage of snapshots. The value oss indicates that snapshots are stored in Object Storage Service (OSS).

    compress

    Specifies whether the data compression feature is enabled for snapshots. Valid values:

    • true: enables the data compression feature for snapshots. This feature applies only to index metadata, including the mappings and settings of indexes.

    • false: disables the data compression feature for snapshots. The default value is false.

    base_path

    The location of snapshots in OSS.

    endpoint

    The endpoint of the OSS bucket that stores snapshots.

  • Run the following command to query the information about all snapshots stored in the aliyun_auto_snapshot repository:

    GET _snapshot/aliyun_auto_snapshot/_all

    If the command is successfully run, the following result is returned:

    {
      "snapshots": [
        {
          "snapshot": "es-cn-09k2053us0003****_20210117030003",
          "uuid": "vIdSCkthTeGa0nSj4D****",
          "version_id": 5050399,
          "version": "5.5.3",
          "indices": [
            ".kibana"
          ],
          "state": "SUCCESS",
          "start_time": "2018-06-28T01:22:39.609Z",
          "start_time_in_millis": 1530148959609,
          "end_time": "2018-06-28T01:22:39.923Z",
          "end_time_in_millis": 1530148959923,
          "duration_in_millis": 314,
          "failures": [],
          "shards": {
            "total": 1,
            "failed": 0,
            "successful": 1
          }
        },
        {
          "snapshot": "es-cn-09k2053us0003****_20210118030004",
          "uuid": "XKO_Uwz_Qu6mZrU3Am****",
          "version_id": 5050399,
          "version": "5.5.3",
          "indices": [
            ".kibana"
          ],
          "state": "SUCCESS",
          "start_time": "2018-06-28T01:25:00.764Z",
          "start_time_in_millis": 1530149100764,
          "end_time": "2018-06-28T01:25:01.482Z",
          "end_time_in_millis": 1530149101482,
          "duration_in_millis": 718,
          "failures": [],
          "shards": {
            "total": 1,
            "failed": 0,
            "successful": 1
          }
        }
      ]
    }
    Important

    The system uses the system time of the current region to create snapshots. However, the time in the returned result is in UTC. A time zone difference exists between the system time and UTC. You can convert the time in the returned result based on the time zone difference. For example, the time zone difference between the system time of the China (Beijing) region and UTC is 8 hours. In this case, the system time of the China (Beijing) region is UTC+0800.

    In the returned result, the state parameter indicates the status of a snapshot. A snapshot can be in one of the following states.

    State

    Description

    IN_PROGRESS

    The snapshot is being created.

    SUCCESS

    The snapshot is created, and all shards are stored.

    FAILED

    The snapshot fails to be created because some shards cannot be stored.

    PARTIAL

    The snapshot is created, but at least one shard fails to be stored.

    INCOMPATIBLE

    The snapshot version is incompatible with the cluster version.

    Automatic snapshots also support the following default parameters that are not displayed.

    Parameter

    Description

    max_snapshot_bytes_per_sec

    The maximum number of bytes that can be backed up for a single node per second. Default value: 40mb.

    max_restore_bytes_per_sec

    The maximum number of bytes that can be restored to a single node per second. Default value: 40mb.

    chunk_size

    During snapshot creation, a large index is divided into multiple parts. This parameter specifies the maximum size of each part. Example values: 1g, 10m, and 5k. Default value: null. This value indicates that the size of each part is not limited.

  • Run the following command to query the information about a specific automatic snapshot stored in the aliyun_auto_snapshot repository:

    GET _snapshot/aliyun_auto_snapshot/<snapshot>/_status

    Replace <snapshot> with the name of the automatic snapshot, such as es-cn-09k2053us0003****_20210118030004. You can query the name of a snapshot by running the command that is used to query all snapshots.

    If the command is successfully run, the following result is returned:

    {
      "snapshots": [
        {
          "snapshot": "es-cn-09k2053us0003****_20210118030004",
          "repository": "aliyun_auto_snapshot",
          "uuid": "XKO_Uwz_Qu6mZrU3Am****",
          "state": "SUCCESS",
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "number_of_files": 4,
            "processed_files": 4,
            "total_size_in_bytes": 3296,
            "processed_size_in_bytes": 3296,
            "start_time_in_millis": 1530148959688,
            "time_in_millis": 77
          },
          "indices": {
            ".kibana": {
              "shards_stats": {
                "initializing": 0,
                "started": 0,
                "finalizing": 0,
                "done": 1,
                "failed": 0,
                "total": 1
              },
              "stats": {
                "number_of_files": 4,
                "processed_files": 4,
                "total_size_in_bytes": 3296,
                "processed_size_in_bytes": 3296,
                "start_time_in_millis": 1530148959688,
                "time_in_millis": 77
              },
              "shards": {
                "0": {
                  "stage": "DONE",
                  "stats": {
                    "number_of_files": 4,
                    "processed_files": 4,
                    "total_size_in_bytes": 3296,
                    "processed_size_in_bytes": 3296,
                    "start_time_in_millis": 1530148959688,
                    "time_in_millis": 77
                  }
                }
              }
            }
          }
        }
      ]
    }

Restore data from automatic snapshots

After automatic snapshots are created, you can call the _restore API to restore data from the snapshots to the cluster for which the snapshots are created.

Important

Indexes whose names start with a period (.) are system indexes. We recommend that you do not restore these indexes. If you restore these indexes, you may fail to access the Kibana console.

In the Kibana console, run one of the following commands to restore indexes from a specific snapshot:

  • Restore all indexes from a specific snapshot that is stored in the aliyun_auto_snapshot repository. The restoration is performed in the background.

    POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore

    Replace <snapshot> with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.

  • Restore all indexes from a specific snapshot that is stored in the aliyun_auto_snapshot repository, and wait until the restoration is complete.

    The _restore API asynchronously runs restoration tasks. After the system confirms that data restoration can be performed, it immediately returns a response and performs restoration in the background. You can configure the wait_for_completion parameter in the command to enable the system to return a response only after the restoration is complete.

    POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore?wait_for_completion=true

    Replace <snapshot> with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.

  • Restore a specific index from a specific snapshot that is stored in the aliyun_auto_snapshot repository, and rename the restored index. The restoration is performed in the background.

    POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore
    {
    "indices": "index_1",
    "rename_pattern": "index_(.+)",
    "rename_replacement": "restored_index_$1"
    }

    Parameter

    Description

    <snapshot>

    Replace it with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.

    indices

    The name of the index that you want to restore.

    rename_pattern

    The regular expression that is used to match the name of the index that you want to restore. This parameter is optional.

    rename_replacement

    The regular expression that is used to rename the matched index. This parameter is optional.

Disable the Auto Snapshot feature

  1. Log on to the Alibaba Cloud Elasticsearch console.
  2. In the left-side navigation pane, click Elasticsearch Clusters.
  3. Navigate to the desired cluster.
    1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
    2. On the Elasticsearch Clusters page, find the cluster and click its ID.
  4. In the left-side navigation pane of the page that appears, click Snapshots.

  5. In the Snapshots (Free Trial) section, turn off Auto Snapshot.