Reuse user and visualization information across Elasticsearch clusters - Elasticsearch

If you want to reuse information of a user, such as the roles assigned to the user, or information of visualizations in a source Elasticsearch cluster for a destination Elasticsearch cluster, you can create manual snapshots to back up the information, and restore the information from the manual snapshots to the destination Elasticsearch cluster.

Prerequisites

Object Storage Service (OSS) is activated. An OSS bucket whose storage class is Standard is created. For more information, see Activate OSS and Create a bucket.
Two Alibaba Cloud Elasticsearch clusters are created in the region in which the OSS bucket resides. One Elasticsearch cluster is used as the source cluster, and the other is used as the destination cluster. For information about the compatibility between Elasticsearch clusters of different versions in the aspect of data restoration from snapshots, see Index compatibility.
Note
- The operations for restoring data from snapshots created for an Elasticsearch V8.X cluster are different from the operations for restoring data from snapshots created for an Elasticsearch V7.X cluster. This topic describes how to restore data from a snapshot created for an Elasticsearch V7.10 cluster to another Elasticsearch V7.10 cluster and how to restore data from a snapshot created for an Elasticsearch V8.5 cluster to another Elasticsearch V8.5 cluster.
- For information about the compatibility between Elasticsearch clusters of different versions in the aspect of data restoration from snapshots, see Index compatibility.

Precautions

You can run the code provided in this topic in the Kibana consoles of your Elasticsearch clusters.
You can use only the _features API to back up and restore system indexes and system data streams for Elasticsearch clusters of V8.0 and later. For more information about the _features API, see Snapshot and restore.

Reuse user information including the roles assigned to a user in a source cluster for a destination cluster

Step 1: Prepare the environment

Log on to the Kibana console of the source Elasticsearch cluster. For more information, see Log on to the Kibana console.
Create a role named cj_test_role and a user named kara in the source Elasticsearch cluster, and assign the cj_test_role role to the kara user. For more information, see Use the RBAC mechanism provided by Elasticsearch X-Pack to implement access control.

Step 2: Create a snapshot to back up data in the source Elasticsearch cluster

Log on to the Kibana console of the source Elasticsearch cluster. For more information, see Log on to the Kibana console.

Create a repository named my_backup in the source Elasticsearch cluster.

Create a repository for a cluster in the cloud.

PUT _snapshot/my_backup/
{
    "type": "oss",
    "settings": {
        "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "xxxxxx",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

Create a repository for a self-managed V8.X cluster. In this case, you must install the elasticsearch-repository-oss plug-in on the cluster. For more information, see Install the elasticsearch-repository-oss plug-in.

Note

For more information about the plug-in, see elasticsearch-repository-oss.

PUT /_snapshot/my_backup
{
    "type": "oss",
    "settings": {
        "oss.client.endpoint": "oss-cn-shanghai.aliyuncs.com",
        "oss.client.access_key_id": "xxx",
        "oss.client.secret_access_key": "xxx",
        "oss.client.bucket": "xxxxxx",
        "oss.client.base_path":"snapshot/",
        "oss.client.compress": true
    }
}

Parameter	Description
endpoint	The internal endpoint of the OSS bucket. For more information about how to obtain the endpoint, see Regions and endpoints.
access_key_id	The AccessKey ID of your account. For more information about how to obtain the AccessKey ID, see Obtain an AccessKey pair.
secret_access_key	The AccessKey secret of your account. For more information about how to obtain the AccessKey secret, see Obtain an AccessKey pair.
bucket	The name of the OSS bucket. For more information about how to obtain the name, see Create a bucket.
compress	Specifies whether to enable the data compression feature for snapshots. Valid values: true: enables the data compression feature for snapshots. This feature applies only to index metadata, including the mappings and settings of indexes. false: disables the data compression feature for snapshots. The default value is false.
chunk_size	If you want to upload large volumes of data to the OSS bucket, you can upload the data in multiple parts. In this case, you can use this parameter to set the size of each part. If the size of a part reaches the value of this parameter, the excess data is distributed to another part.
base_path	The start location of the repository. The default value is the root directory. You can specify the directory where specific snapshots are stored. Example: snapshot/myindex/.

Create a snapshot for the source Elasticsearch cluster.
- If the source Elasticsearch cluster is of V7.10, run the following command to create a snapshot named snapshot_1 for the cluster.
  Note
  The information of a user, such as the roles assigned to the user, in an Elasticsearch V7.10 cluster is managed by using the .kibana* and .security* system indexes.
  For information about the parameters in the command that is used to create a snapshot for an Elasticsearch V7.10 cluster, see Restore snapshot API.
```
PUT _snapshot/my_backup/snapshot_1
{
 "indices": ".kibana*,.security*",
 "ignore_unavailable": true,
 "include_global_state": true
}
```
- If the source Elasticsearch cluster is of V8.5, run the following command to create a snapshot named snapshot_1 for the indexes that store information of the user, such as the role assigned to the user, in the cluster.
  Note
  For information about the parameters in the command that is used to create a snapshot for an Elasticsearch V8.5 cluster, see Create snapshot API.
```
PUT _snapshot/my_backup/snapshot_1
{
 "indices": "-*",
 "ignore_unavailable": true,
 "include_global_state": true,
 "feature_states": [
 "security"
 ]
}
```
Check whether the snapshot is successfully created. For example, you can run the following command to query the details of the snapshot_1 snapshot in the my_backup repository.
```
GET _snapshot/my_backup/snapshot_1
```

Step 3: Restore data from the snapshot

Log on to the Kibana console of the destination Elasticsearch cluster. For more information, see Log on to the Kibana console.

Create a repository named my_restore in the destination Elasticsearch cluster.

PUT _snapshot/my_restore
{
 "type": "oss",
 "settings": {
 "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
 "access_key_id": "[your_access_key_id]",
 "secret_access_key": "[your_secret_access_key]",
 "bucket": "[your_bucket_name]",
 "compress": true,
 "chunk_size": "500mb",
 "base_path": "snapshot/"
 }
}

Restore data from the snapshot that is created for the source Elasticsearch cluster to the destination Elasticsearch cluster.
- If the source Elasticsearch cluster is of V7.10, perform the following operations to restore data from the snapshot created for the cluster.
  Note
  If you want to restore data from a snapshot created for an Elasticsearch cluster of V7.10 or earlier to a destination Elasticsearch cluster, you must disable the .kibana* and .security* system indexes in the destination Elasticsearch cluster. After data is restored from the snapshot, you must enable these indexes at the earliest opportunity.
  - Disabling system indexes may cause the features of the Elasticsearch cluster to be unavailable or limited. For example, disabling of .kibana* system indexes may cause the Kibana console to be unavailable, and disabling of .security* system indexes may affect the data security of the Elasticsearch cluster.
  - We recommend that you back up data in these system indexes in advance. This way, you can restore the data if necessary.
  - We recommend that you disable these indexes in the destination Elasticsearch cluster during off-peak hours or when the cluster is not providing services.
  - If you cannot disable these indexes in the Kibana console of the destination Elasticsearch cluster, we recommend that you use a terminal to disable these indexes. For more information, see Curl commands that you can use to manage an Elasticsearch cluster.
  1. Disable the .kibana* and .security* system indexes in the destination Elasticsearch cluster.
```
POST /<index name>/_close
```
  2. Restore data from the snapshot_1 snapshot to the destination Elasticsearch cluster.
```
POST /_snapshot/my_restore/snapshot_1/_restore
{
 "indices": ".kibana*,.security*",
 "ignore_unavailable": true,
 "include_global_state": false
}
```
  3. Enable the .kibana* and .security* system indexes in the destination Elasticsearch cluster.
```
POST /<index name>/_open
```
- If the source Elasticsearch cluster is of V8.5, run the following command in the Kibana console of the destination Elasticsearch cluster to restore data from the snapshot_1 snapshot to the destination Elasticsearch cluster.
```
POST _snapshot/my_restore/snapshot_1/_restore
{
 "feature_states": [ "security" ],
 "include_global_state": false, 
 "indices": "-*" 
}
```
Check whether the user information is reused for the destination Elasticsearch cluster.
1. In the upper-left corner of the Kibana console of the destination Elasticsearch cluster, click the icon. In the left-side navigation pane, choose Management > Stack Management.
2. In the left-side navigation pane, click Users. On the Users page, check whether the kara user to which the cj_test_role role is assigned exists. If the user exists, information of the kara user is reused for the destination Elasticsearch cluster.

Reuse visualizations in a source Elasticsearch cluster for a destination Elasticsearch cluster