After your Kibana service is restarted due to exceptions, .kibana_{num} indexes for the service are reset. In this case, you must restore the indexes to restore data. In .kibana_{num}, {num} is a variable. For example, the .kibana_1 and .kibana_2 indexes may exist. This topic describes how to use a snapshot or call the reindex API to restore a .kibana_{num} index. In this topic, the .kibana_1 index is used.

Background information

.kibana_{num} indexes are used to store Kibana configuration information, such as dashboard configurations and index patterns. By default, Kibana reads data from .kibana_{num} indexes based on the alias .kibana. If multiple .kibana_{num} indexes exist, only one of the indexes uses the alias .kibana.

In some complex business scenarios, Kibana configuration information stored in a .kibana_{num} index may be reset. For example, if the structure of the .kibana_1 index is damaged or an error occurs when you access the Kibana service, you need to delete the .kibana_1 index and restart your Kibana service. If a configuration exception occurs on a scheduling task, the .kibana_1 index may be deleted, and the Kibana service may be restarted. In these scenarios, after the Kibana service is restarted, .kibana_task_manager_1 automatically generates a new .kibana_1 index that uses the alias .kibana, and Kibana configuration information is reset in the new .kibana_1 index. The information includes index patterns, dashboard configurations, and user roles.

You can use one of the following methods to restore the .kibana_1 index:
  • Use a snapshot to directly restore the .kibana_1 index.
  • Restore data in a snapshot to a backup index and reindex the data in the backup index to the .kibana_1 index.
  • Manually create index patterns or user roles in the Kibana service.

Prerequisites

  • The Auto Snapshot feature is enabled for your Elasticsearch cluster, or manual snapshots are created for system indexes in your Elasticsearch cluster. The structure of the .kibana_1 index in snapshots is normal.
  • The alias .kibana is used for the .kibana_1 index. If the .kibana_1 index does not exist, we recommend that you restart your Kibana service to generate the .kibana_1 index.

Procedure

Use a snapshot to restore the .kibana_1 index

  1. Log on to the Kibana console of your Elasticsearch cluster and go to the homepage of the Kibana console as prompted.
    For more information about how to log on to the Kibana console, see Log on to the Kibana console.
    Note In this example, an Elasticsearch V7.10.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.
  2. In the upper-right corner of the page that appears, click Dev tools.
  3. On the Console tab of the page that appears, run the following command to query the information about automatic snapshots stored in the related repository: 
    GET _cat/snapshots/aliyun_auto_snapshot?v

    aliyun_auto_snapshot is the name of the repository that stores automatic snapshots. If you use the Auto Snapshot feature provided by Alibaba Cloud Elasticsearch to periodically create automatic snapshots for data in your Elasticsearch cluster, the name of the repository that stores the snapshots is fixed as aliyun_auto_snapshot. If you manually create snapshots for data in your Elasticsearch cluster, you must replace aliyun_auto_snapshot with the name of the repository that you use.

    If the command is successfully run, the system returns the information about all snapshots stored in the repository, such as the IDs and status of the snapshots.
    Note We recommend that you record the ID of the latest snapshot created before an error occurs on the Kibana service for subsequent use.
    Query information about snapshots in the repository
  4. Run the following command to query the details of the latest snapshot and check whether the .kibana_1 index exists in the Elasticsearch cluster: 
    GET _snapshot/aliyun_auto_snapshot/<yourSnapshotId>

    The <yourSnapshotId> parameter specifies the ID of the latest snapshot created before an error occurs on the Kibana service. Set this parameter to the ID that you obtained in Step 3. In this example, this parameter is set to es-cn-m7r23wodb006n****_20220303020235.

    If the command is successfully run, the result shown in the following figure is returned. You can proceed to subsequent steps only if the result contains the .kibana_1 index. Check whether the .kibana_1 index exists
  5. Run the following command to set the status of the .kibana_1 index to close:
    Important If you cannot log on to the Kibana console after you set the status of the .kibana_1 index to close and then restart the Kibana service, you must run a cURL command to restore the. kibana_1 index. For more information, see FAQ. 
    POST /.kibana_1/_close
    If the command is successfully run, the following result is returned: 
    {
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "indices" : {
    ".kibana_1" : {
      "closed" : true
    }
  }
}
  6. Run the following command to restore the .kibana_1 index from the snapshot: 
    POST _snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore
{
   "indices": ".kibana_1",
   "rename_pattern": ".kibana_1",
   "rename_replacement": ".kibana_1"
}
    Parameter Description
    <yourSnapshotId> The ID of the latest automatic snapshot created before an error occurs on the Kibana service. In this example, set this parameter to the ID that you obtained in Step 3.
    indices The name of the index that you want to restore.
    rename_pattern The regular expression that is used to match the name of the index that you want to restore. This parameter is optional.
    rename_replacement The regular expression that is used to rename the matched index. This parameter is optional.
    If the command is successfully run, "accepted" : true is returned.
    Important If your Elasticsearch cluster is of a version other than V7.10, an error similar to index_closed_exception index=".kibana_1" may be reported when you use a snapshot to restore the .kibana_1 index. You can troubleshoot the error based on the instructions provided in FAQ.
  7. View the restoration result.
    Perform the following steps to check whether index patterns that are created in the Kibana service are restored. If the index patterns are restored, the .kibana_1 index is restored from the snapshot.
    1. In the upper-left corner of the Kibana console, click the Show the left-side navigation pane icon.
    2. In the left-side navigation pane, choose Management > Stack Management.
    3. In the Kibana section, click Index Patterns.
    4. On the Index patterns page, click a pattern whose name starts with kibana_ to check whether data is restored.
      View the restoration result

Call the reindex API to restore the .kibana_1 index

  1. Query the details of the latest snapshot and check whether the .kibana_1 index exists in the Elasticsearch cluster.
    For more information, see Step 1 to Step 4 in Use a snapshot to restore the .kibana_1 index.
    Note You can proceed to subsequent steps only if the .kibana_1 index exists in the Elasticsearch cluster.
  2. Run the following command to restore the .kibana_1 index in the snapshot to a backup index. In this example, the backup index is kibana123. 
    POST _snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore
{
   "indices": ".kibana_1",
   "rename_pattern": ".kibana_1",
   "rename_replacement": "kibana123"
}
    Parameter Description
    <yourSnapshotId> The ID of the latest automatic snapshot created before an error occurs on the Kibana service. In this example, set this parameter to the ID that you obtained in Step 1.
    indices The name of the index that you want to restore.
    rename_pattern The regular expression that is used to match the name of the index that you want to restore. This parameter is optional.
    rename_replacement The name of the backup index.

    If the command is successfully run, "accepted" : true is returned.

  3. Call the reindex API to reindex data in the backup index to the current .kibana_1 index. 
    POST _reindex
{
  "source": {
    "index": "kibana123"
  },
  "dest": {
    "index": ".kibana_1"
  }
}
    If the command is successfully run, the following result is returned: 
    {
  "took" : 731,
  "timed_out" : false,
  "total" : 33,
  "updated" : 33,
  "created" : 0,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}
  4. Delete the backup index. In this example, the backup index is kibana123. 
    DELETE kibana123

    If the command is successfully run, "accepted" : true is returned.

  5. View the restoration result.
    Check whether index patterns that are created in the Kibana service are restored. If the index patterns are restored, the .kibana_1 index is restored. For more information, see Step 7 in Use a snapshot to restore the .kibana_1 index. View the restoration result

FAQ

  • Q: When I use a snapshot to restore the .kibana_1 index that uses the alias .kibana, the system displays an error message indicating that the alias of the .kibana_1 index conflicts with the name of the .kibana index. What do I do?

    A: After the .kibana_1 index is deleted, Elasticsearch cannot read data based on the alias .kibana. As a result, Elasticsearch creates a .kibana index. The .kibana_1 index in the snapshot uses .kibana as its alias. When you restore the .kibana_1 index, the system displays an error message indicating that an alias conflict occurs. In this case, we recommend that you delete the .kibana index before you restore the .kibana_1 index from a snapshot.

  • Q: When I run the related command for an Elasticsearch cluster of a version other than V7.10 to restore the .kibana_1 index from a snapshot, the error message shown in the following figure is reported. What do I do? Error
    A: The error message is reported because the .kibana_1 index is in the close state. If an index is in the close state, the error message is reported when you run any command for the index in the Kibana console. We recommend that you use an Elastic Compute Service (ECS) instance to connect to the Elasticsearch cluster and run a cURL command to restore the index. For more information, see Use curl commands and API operations to manage an Alibaba Cloud Elasticsearch cluster. For example, you can run the following cURL command to restore the .kibana_1 index: 
    curl -u <user>:<password> -XPOST "http://<host>:<port>/_snapshot/aliyun_auto_snapshot/<yourSnapshotId>/_restore" -H 'Content-Type: application/json' -d'{"indices": ".kibana_1","rename_pattern": ".kibana_1","rename_replacement": ".kibana_1"}'