Alibaba Cloud Elasticsearch provides the auto snapshot feature. This feature allows you to specify the creation interval and time for snapshots. After you specify them, the system automatically creates snapshots for your Elasticsearch cluster based on the specified interval and time to ensure data security. These snapshots are called automatic snapshots. You can then restore data from the snapshots to the cluster. This topic describes how to use the auto snapshot feature, view the information of automatic snapshots, and restore data from the automatic snapshots.

Precautions

  • The first snapshot is a full copy of the data in a cluster. Subsequent snapshots store only incremental data. Therefore, it requires a longer time to create the first snapshot than a subsequent snapshot. The specific time depends on the data volume.
  • Snapshots store only index data. The following information of an Elasticsearch cluster is not stored in snapshots: monitoring data (such as indexes whose names start with .monitoring or .security_audit), metadata, translogs, configurations, software packages, built-in and custom plug-ins, and logs.
  • The system stores only the automatic snapshots that are created during the last seven days.
  • You can restore data from automatic snapshots only to the Elasticsearch cluster for which the snapshots are created. For more information about how to restore data to a cluster other than the original cluster, see Create manual snapshots and restore data from manual snapshots or Configure a shared OSS repository.

Enable the auto snapshot feature

  1. Log on to the Elasticsearch console.
  2. In the left-side navigation pane, click Elasticsearch Clusters.
  3. Navigate to the desired cluster.
    1. In the top navigation bar, select a resource group and a region.
    2. In the left-side navigation pane, click Elasticsearch Clusters. On the Elasticsearch Clusters page, find the desired cluster and click its ID.
  4. In the left-side navigation pane of the page that appears, click Snapshots.
  5. In the Snapshots (Free Trial) section, turn on Auto Snapshot.
  6. On the right side of the page, click Modify Configuration.
    Notice To ensure system security, the creation interval and time of automatic snapshots for Advanced Edition clusters are generated by the system and cannot be changed.
  7. In the Auto Snapshot Configuration panel, set Frequency.
    Set an interval for automatic snapshot creation
    Frequency Description
    Every 30 Minutes The system creates snapshots at 30-minute intervals.
    Daily The system creates snapshots every day. You can customize creation time.
    Custom The system creates snapshots based on the interval and time you specify.
    Notice The system uses the system time of the region where your cluster resides to create snapshots.
  8. Click OK.

View the information of automatic snapshots

After you enable the auto snapshot feature, you can log on to the Kibana console of your cluster and call the _snapshot API to view the information of the automatic snapshots of the cluster. For more information about how to log on to the Kibana console of a cluster, see Log on to the Kibana console.

  • Run the following command to query snapshot repositories:
    GET _snapshot
    If the command is successfully run, the following result is returned:
    {
      "aliyun_auto_snapshot" : {
        "type" : "oss",
        "settings" : {
          "compress" : "true",
          "base_path" : "cf95b9-185320276651****/es-cn-09k2053us0003****",
          "endpoint" : "http://oss-cn-hangzhou-internal.aliyuncs.com"
        }
      }
    }
                                
    Parameter Description
    aliyun_auto_snapshot The name of the repository. An automatic snapshot repository is created when the first snapshot is created. The name of the repository that is used to store the automatic snapshots of Alibaba Cloud Elasticsearch clusters is fixed to aliyun_auto_snapshot.
    Note A repository can store multiple snapshots, each of which can contain the data of all indexes, some indexes, or a single index.
    type The storage of snapshots. The value oss indicates that snapshots are stored in Object Storage Service (OSS).
    compress Specifies whether the data compression feature is enabled for snapshots. Valid values:
    • true: indicates that the data compression feature is enabled for snapshots. This feature applies only to index metadata, including the mappings and settings of indexes.
    • false: indicates that the data compression feature is disabled for snapshots. The default value is false.
    base_path The location of snapshots in OSS.
    endpoint The endpoint of the OSS bucket that stores snapshots.
  • Run the following command to query the information of all the snapshots stored in the aliyun_auto_snapshot repository:
    GET _snapshot/aliyun_auto_snapshot/_all
    If the command is successfully run, the following result is returned:
    {
      "snapshots": [
        {
          "snapshot": "es-cn-09k2053us0003****_20210117030003",
          "uuid": "vIdSCkthTeGa0nSj4D****",
          "version_id": 5050399,
          "version": "5.5.3",
          "indices": [
            ".kibana"
          ],
          "state": "SUCCESS",
          "start_time": "2018-06-28T01:22:39.609Z",
          "start_time_in_millis": 1530148959609,
          "end_time": "2018-06-28T01:22:39.923Z",
          "end_time_in_millis": 1530148959923,
          "duration_in_millis": 314,
          "failures": [],
          "shards": {
            "total": 1,
            "failed": 0,
            "successful": 1
          }
        },
        {
          "snapshot": "es-cn-09k2053us0003****_20210118030004",
          "uuid": "XKO_Uwz_Qu6mZrU3Am****",
          "version_id": 5050399,
          "version": "5.5.3",
          "indices": [
            ".kibana"
          ],
          "state": "SUCCESS",
          "start_time": "2018-06-28T01:25:00.764Z",
          "start_time_in_millis": 1530149100764,
          "end_time": "2018-06-28T01:25:01.482Z",
          "end_time_in_millis": 1530149101482,
          "duration_in_millis": 718,
          "failures": [],
          "shards": {
            "total": 1,
            "failed": 0,
            "successful": 1
          }
        }
      ]
    }
    Notice The system uses the system time of the current region to create snapshots. However, the time in the returned result is in UTC. A time zone difference exists between the system time and UTC. You can convert the time in the returned result based on the time zone difference. For example, if the time zone difference between the system time of the China (Beijing) region and UTC is 8 hours, the system time of the China (Beijing) region is UTC+0080.
    In the returned result, the state field indicates the status of a snapshot. A snapshot can be in one of the following states.
    State Description
    IN_PROGRESS The snapshot is being created.
    SUCCESS The snapshot is created, and all shards are stored.
    FAILED The snapshot fails to be created because some shards cannot be stored.
    PARTIAL The snapshot is created, but at least one shard fails to be stored.
    INCOMPATIBLE The snapshot version is incompatible with the cluster version.
    The auto snapshot feature also supports the following default parameters that are not displayed.
    Parameter Description
    max_snapshot_bytes_per_sec The maximum number of bytes that can be backed up in a snapshot per second. Default value: 40mb.
    max_restore_bytes_per_sec The maximum number of bytes that can be restored from a snapshot per second. Default value: 40mb.
    chunk_size During snapshot creation, a large index is divided into multiple parts. This parameter specifies the maximum size of each part. Example values: 1g, 10m, and 5k. Default value: null. This value indicates that the size of each part is not limited.
  • Run the following command to query the information of a specific automatic snapshot stored in the aliyun_auto_snapshot repository:
    GET _snapshot/aliyun_auto_snapshot/<snapshot>/_status

    Replace <snapshot> with the name of the automatic snapshot, such as es-cn-09k2053us0003****_20210118030004. You can query the name of a snapshot by running the command that is used to query all snapshots.

    If the command is successfully run, the following result is returned:
    {
      "snapshots": [
        {
          "snapshot": "es-cn-09k2053us0003****_20210118030004",
          "repository": "aliyun_auto_snapshot",
          "uuid": "XKO_Uwz_Qu6mZrU3Am****",
          "state": "SUCCESS",
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "number_of_files": 4,
            "processed_files": 4,
            "total_size_in_bytes": 3296,
            "processed_size_in_bytes": 3296,
            "start_time_in_millis": 1530148959688,
            "time_in_millis": 77
          },
          "indices": {
            ".kibana": {
              "shards_stats": {
                "initializing": 0,
                "started": 0,
                "finalizing": 0,
                "done": 1,
                "failed": 0,
                "total": 1
              },
              "stats": {
                "number_of_files": 4,
                "processed_files": 4,
                "total_size_in_bytes": 3296,
                "processed_size_in_bytes": 3296,
                "start_time_in_millis": 1530148959688,
                "time_in_millis": 77
              },
              "shards": {
                "0": {
                  "stage": "DONE",
                  "stats": {
                    "number_of_files": 4,
                    "processed_files": 4,
                    "total_size_in_bytes": 3296,
                    "processed_size_in_bytes": 3296,
                    "start_time_in_millis": 1530148959688,
                    "time_in_millis": 77
                  }
                }
              }
            }
          }
        }
      ]
    }

Restore data from automatic snapshots

After automatic snapshots are created, you can call the _restore API to restore data from the snapshots to the cluster for which the snapshots are created.
Note We recommend that you do not restore system indexes whose names start with a period (.). If you restore these indexes, you may fail to access the Kibana console.
In the Kibana console, run one of the following commands to restore indexes from a specific snapshot:
  • Restore all indexes from a specific snapshot that is stored in the aliyun_auto_snapshot repository. The restoration is performed in the background.
    POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore

    Replace <snapshot> with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.

  • Restore all indexes from a specific snapshot that is stored in the aliyun_auto_snapshot repository, and wait until the restoration is complete.
    The _restore API asynchronously runs restoration tasks. After the system confirms that data restoration can be performed, it immediately returns a response and performs restoration in the background. You can specify the wait_for_completion parameter in the command to enable the system to return a response only after the restoration is complete.
    POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore?wait_for_completion=true

    Replace <snapshot> with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.

  • Restore specified indexes from a specific snapshot that is stored in the aliyun_auto_snapshot repository, and rename the restored indexes. The restoration is performed in the background.
    POST _snapshot/aliyun_auto_snapshot/<snapshot>/_restore
    {
    "indices": "index_1",
    "rename_pattern": "index_(.+)",
    "rename_replacement": "restored_index_$1"
    }
    Parameter Description
    <snapshot> Replace it with the name of the specific snapshot, such as es-cn-abcdefghij****_20180627091600.
    indices The name of the index that you want to restore.
    rename_pattern Optional. This parameter specifies the regular expression that is used to match the name of the index you want to restore.
    rename_replacement Optional. This parameter specifies the regular expression that is used to rename a matching index.