Restore a .kibana_{num} index from a snapshot - Elasticsearch

When a Kibana cluster is reset, its .kibana_{num} configuration indices ({num} is a variable, such as .kibana_1 or .kibana_2) are also reset. This topic uses .kibana_1 as an example to show how to restore a .kibana_{num} index from a snapshot or by reindexing.

Background

The .kibana_{num} index stores Kibana configurations such as dashboards and index patterns. Kibana reads data through the .kibana alias, which points to only one .kibana_{num} index even when multiple exist. Only that index uses the .kibana alias.

If the .kibana_1 index is corrupted and you need to delete the .kibana_1 index to reset Kibana, or if a misconfigured scheduled task deletes the .kibana_1 index, .kibana_task_manager_1 generates a new .kibana_1 index bound to the .kibana alias. All previously configured index patterns, dashboards, and user roles are lost.

You can restore the .kibana_1 index using these methods:

Restore directly from a snapshot.
Restore to a backup index, then reindex the data into the .kibana_1 index.
Manually rebuild Kibana index patterns or user roles.

Prerequisites

Automatic snapshot backup is enabled, or you have manually backed up system indices. The snapshot must contain a healthy .kibana_1 index.
The .kibana alias exists and points to the .kibana_1 index. If the .kibana_1 index does not exist, restart Kibana to initialize a new .kibana_1 index.

Procedure

Snapshot restore

Log on to the Kibana console of your Elasticsearch cluster and go to the Kibana homepage.
In the left navigation menu, click Dev tools.

On the Console tab, run the following command to list all snapshots in the repository.

GET _cat/snapshots/aliyun_auto_snapshot?v

aliyun_auto_snapshot is the default repository name for Alibaba Cloud Elasticsearch automatic backups (always aliyun_auto_snapshot). If you use manual backups, replace aliyun_auto_snapshot with your repository name.

The output lists all snapshots with their id and status.

Note

Record the id of the latest snapshot created before the issue occurred.

Sample output:

GET _cat/snapshots/aliyun_auto_snapshot?v

   id                              status  start_epoch start_time end_epoch  end_time duration indices successful_shards failed_shards total_shards
1  es-cn-m7r23wodb0xxx 20220303020235 SUCCESS 1646244156  18:02:36   1646244168 18:02:48    12.6s      47            66             0           66
2  es-cn-m7r23wodb0xxx 20220304020204 SUCCESS 1646330525  18:02:05   1646330541 18:02:21    16.2s      47            66             0           66
3  es-cn-m7r23wodb0xxx 20220305020209 SUCCESS 1646416929  18:02:09   1646416946 18:02:26      17s      47            66             0           66
4  es-cn-m7r23wodb0xxx 20220306020206 SUCCESS 1646503327  18:02:07   1646503346 18:02:26    18.8s      47            66             0           66
5  es-cn-m7r23wodb0xxx 20220307020200 SUCCESS 1646589721  18:02:01   1646589743 18:02:23    21.8s      47            66             0           66
6  es-cn-m7r23wodb0xxx 20220308020218 SUCCESS 1646676142  18:02:22   1646676155 18:02:35      13s      47            66             0           66
7  es-cn-m7r23wodb0xxx 20220309020211 SUCCESS 1646762532  18:02:12   1646762539 18:02:19       7s      47            66             0           66
8  es-cn-m7r23wodb0xxx 20220310020210 SUCCESS 1646848931  18:02:11   1646848946 18:02:26    14.6s      47            66             0           66
9  es-cn-m7r23wodb0xxx 20220311020208 SUCCESS 1646935329  18:02:09   1646935342 18:02:22    13.8s      47            66             0           66
10 es-cn-m7r23wodb0xxx 20220312020212 SUCCESS 1647021733  18:02:13   1647021749 18:02:29      16s      47            66             0           66

Verify that the snapshot contains the .kibana_1 index.

GET _snapshot/aliyun_auto_snapshot/<yourSnapshotId>

<yourSnapshotId>: The snapshot id obtained in the previous step, for example, es-cn-m7r23wodb006n****_20220303020235.

Proceed only if the output contains the .kibana_1 index.

In Kibana Dev Tools, run the following commands and confirm that the indices list in the response contains the .kibana_1 index. This indicates that the snapshot can be used to restore Kibana data.

GET _cat/snapshots/aliyun_auto_snapshot?v

GET _snapshot/aliyun_auto_snapshot/es-cn-m7r23wodbxxx_20220303020235

{
  "snapshots" : [
    {
      "snapshot" : "es-cn-m7r23woxxx_20220303020235",
      "uuid" : "AiB8unLlT9aWPoxxx",
      "version_id" : 7100099,
      "version" : "7.10.0",
      "indices" : [
        ".kibana-event-log-7.10.0-000009",
        ".kibana-event-log-7.10.0-000010",
        "zltest",
        ".watches",
        "my-index-000001",
        ".transform-internal-005",
        ".async-search",
        "chinese_news",
        ".watcher-history-12-2022.03.02",
        "kibana_sample_data_logs",
        "zltest14",
        ".transform-notifications-000002",
        "zltest17",
        ".kibana_1",
        ".security-7",
        ".kibana-event-log-7.10.0-000012",
        "zltest16",
        "ilm-history-3-000010",
        ".triggered_watches",
        ".watcher-history-12-2022.02.25",
        "dest",
        "my-index-12345e",
        ".watcher-history-12-2022.02.24",
        "metrics-endpoint.metadata_current_default",
        ...
      ]
    }
  ]
}

Close the .kibana_1 index.
Important
Closing the .kibana_1 index may prevent Kibana login. If this happens, restore the index using a curl command instead. FAQ.
```
POST /.kibana_1/_close
```
Expected output:
```
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "indices" : {
    ".kibana_1" : {
      "closed" : true
    }
  }
}
```

Restore the .kibana_1 index from the snapshot.

POST _snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore
{
   "indices": ".kibana_1",
   "rename_pattern": ".kibana_1",
   "rename_replacement": ".kibana_1"
}

Parameter	Description
<yourSnapshotId>	The snapshot id. Enter the id obtained in Step 3.
indices	The name of the index to restore.
rename_pattern	Optional. A regular expression that matches the name of the index to restore.
rename_replacement	Optional. Renames the matched index as specified.

On success, "accepted" : true is returned.

Important

On instances with a version other than 7.10, if an index_closed_exception index=".kibana_1" error occurs when restoring the .kibana_1 index, see FAQ for a solution.

Verify the restore.

Confirm that Kibana index patterns are restored.
1. In the upper-left corner of the Kibana console, click the icon.
2. In the navigation pane on the left, go to Management > Stack Management.
3. In the Kibana section, click Index Patterns.
4. On the Index Patterns page, click an index pattern that starts with kibana_ to check whether the data is restored.

Reindex restore

Verify that the snapshot contains the .kibana_1 index.

Follow Steps 1 to 4 in Snapshot restore.

Note
Proceed only if the snapshot contains the .kibana_1 index.

Restore the .kibana_1 index from the snapshot to a backup index named kibana123.

POST _snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore
{
   "indices": ".kibana_1",
   "rename_pattern": ".kibana_1",
   "rename_replacement": "kibana123"
}

Parameter	Description
<yourSnapshotId>	The snapshot id. Enter the id obtained in Step 1.
indices	The name of the index to restore.
rename_pattern	Optional. A regular expression that matches the name of the index to restore.
rename_replacement	The name of the backup index.

On success, "accepted" : true is returned.

Reindex data from the backup index to the .kibana_1 index.

POST _reindex
{
  "source": {
    "index": "kibana123"
  },
  "dest": {
    "index": ".kibana_1"
  }
}

Expected output:

{
  "took" : 731,
  "timed_out" : false,
  "total" : 33,
  "updated" : 33,
  "created" : 0,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}

Delete the backup index kibana123.
```
DELETE kibana123
```
On success, "accepted" : true is returned.
Verify the restore.

Confirm that Kibana index patterns are restored. Follow Step 7 in Snapshot restore. On the Index Patterns page in Kibana, confirm that the following three sample data index patterns are successfully restored: kibana_sample_data_ecommerce, kibana_sample_data_flights, and kibana_sample_data_logs.

FAQ

Q: I get an alias conflict between the .kibana index, .kibana alias, and .kibana_1 during snapshot restore. How do I fix this?

A: After .kibana_1 is deleted, Elasticsearch automatically creates a .kibana index. This conflicts with the .kibana_1 index in the snapshot, which also uses .kibana as an alias. Delete the .kibana index before performing the restore.

Q: On a non-7.10 instance, I get an index_closed_exception error when restoring .kibana_1. How do I fix this?

A: This error occurs because the closed .kibana_1 index prevents commands from running in the Kibana console. Connect to Elasticsearch from an ECS instance and run the curl command instead. Connect to a cluster from the command line. Example curl command to restore .kibana_1:

curl -u <user>:<password> -XPOST "http://<host>:<port>/_snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore" -H 'Content-Type: application/json' -d'{"indices": ".kibana_1","rename_pattern": ".kibana_1","rename_replacement": ".kibana_1"}'