Restore a .kibana_{num} index from a snapshot - Elasticsearch

When a Kibana cluster is reset, its .kibana_{num} configuration indices ({num} is a variable, such as .kibana_1 or .kibana_2) are also reset. This topic uses .kibana_1 as an example to show how to restore a .kibana_{num} index from a snapshot or by reindexing.

Background

The .kibana_{num} index stores Kibana configurations such as dashboards and index patterns. Kibana reads data through the .kibana alias, which points to only one .kibana_{num} index even when multiple exist. Only that index uses the .kibana alias.

If the .kibana_1 index is corrupted and you need to delete the .kibana_1 index to reset Kibana, or if a misconfigured scheduled task deletes the .kibana_1 index, .kibana_task_manager_1 generates a new .kibana_1 index bound to the .kibana alias. All previously configured index patterns, dashboards, and user roles are lost.

You can restore the .kibana_1 index using these methods:

Restore directly from a snapshot.
Restore to a backup index, then reindex the data into the .kibana_1 index.
Manually rebuild Kibana index patterns or user roles.

Prerequisites

Automatic snapshot backup is enabled, or you have manually backed up system indices. The snapshot must contain a healthy .kibana_1 index.
The .kibana alias exists and points to the .kibana_1 index. If the .kibana_1 index does not exist, restart Kibana to initialize a new .kibana_1 index.

Procedure

Snapshot restore

Log on to the Kibana console of your Elasticsearch cluster and go to the Kibana homepage.
In the left navigation menu, click Dev tools.
On the Console tab, run the following command to list all snapshots in the repository.
```
GET _cat/snapshots/aliyun_auto_snapshot?v
```
aliyun_auto_snapshot is the default repository name for Alibaba Cloud Elasticsearch automatic backups (always aliyun_auto_snapshot). If you use manual backups, replace aliyun_auto_snapshot with your repository name.

The output lists all snapshots with their id and status.

Note
Record the id of the latest snapshot created before the issue occurred.
Verify that the snapshot contains the .kibana_1 index.
```
GET _snapshot/aliyun_auto_snapshot/<yourSnapshotId>
```
<yourSnapshotId>: The snapshot id obtained in the previous step, for example, es-cn-m7r23wodb006n****_20220303020235.

Proceed only if the output contains the .kibana_1 index.
Close the .kibana_1 index.
Important
Closing the .kibana_1 index may prevent Kibana login. If this happens, restore the index using a curl command instead. FAQ.
```
POST /.kibana_1/_close
```
Expected output:
```
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "indices" : {
    ".kibana_1" : {
      "closed" : true
    }
  }
}
```

Restore the .kibana_1 index from the snapshot.

POST _snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore
{
   "indices": ".kibana_1",
   "rename_pattern": ".kibana_1",
   "rename_replacement": ".kibana_1"
}

Parameter	Description
<yourSnapshotId>	The snapshot id. Enter the id obtained in Step 3.
indices	The name of the index to restore.
rename_pattern	Optional. A regular expression that matches the name of the index to restore.
rename_replacement	Optional. Renames the matched index as specified.

On success, "accepted" : true is returned.

Important

On instances with a version other than 7.10, if an index_closed_exception index=".kibana_1" error occurs when restoring the .kibana_1 index, see FAQ for a solution.

Verify the restore.

Confirm that Kibana index patterns are restored.
1. In the upper-left corner of the Kibana console, click the icon.
2. In the navigation pane on the left, go to Management > Stack Management.
3. In the Kibana section, click Index Patterns.
4. On the Index Patterns page, click an index pattern that starts with kibana_ to check whether the data is restored.

Reindex restore

Verify that the snapshot contains the .kibana_1 index.

Follow Steps 1 to 4 in Snapshot restore.

Note
Proceed only if the snapshot contains the .kibana_1 index.

Restore the .kibana_1 index from the snapshot to a backup index named kibana123.

POST _snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore
{
   "indices": ".kibana_1",
   "rename_pattern": ".kibana_1",
   "rename_replacement": "kibana123"
}

Parameter	Description
<yourSnapshotId>	The snapshot id. Enter the id obtained in Step 1.
indices	The name of the index to restore.
rename_pattern	Optional. A regular expression that matches the name of the index to restore.
rename_replacement	The name of the backup index.

On success, "accepted" : true is returned.

Reindex data from the backup index to the .kibana_1 index.

POST _reindex
{
  "source": {
    "index": "kibana123"
  },
  "dest": {
    "index": ".kibana_1"
  }
}

Expected output:

{
  "took" : 731,
  "timed_out" : false,
  "total" : 33,
  "updated" : 33,
  "created" : 0,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}

Delete the backup index kibana123.
```
DELETE kibana123
```
On success, "accepted" : true is returned.
Verify the restore.

Confirm that Kibana index patterns are restored. Follow Step 7 in Snapshot restore.

FAQ

Q: I get an alias conflict between the .kibana index, .kibana alias, and .kibana_1 during snapshot restore. How do I fix this?

A: After .kibana_1 is deleted, Elasticsearch automatically creates a .kibana index. This conflicts with the .kibana_1 index in the snapshot, which also uses .kibana as an alias. Delete the .kibana index before performing the restore.

Q: On a non-7.10 instance, I get an index_closed_exception error when restoring .kibana_1. How do I fix this?

A: This error occurs because the closed .kibana_1 index prevents commands from running in the Kibana console. Connect to Elasticsearch from an ECS instance and run the curl command instead. Connect to a cluster from the command line. Example curl command to restore .kibana_1:

curl -u <user>:<password> -XPOST "http://<host>:<port>/_snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore" -H 'Content-Type: application/json' -d'{"indices": ".kibana_1","rename_pattern": ".kibana_1","rename_replacement": ".kibana_1"}'