All Products
Search
Document Center

Elasticsearch:Use the CCR feature in disaster recovery scenarios

Last Updated:Mar 01, 2024

If a disaster, such as a service interruption caused by a hardware failure, software error, data center failure, or natural disaster, occurs on your Elasticsearch cluster, you can use the cross-cluster replication (CCR) feature to restore lost data to the cluster.

Background information

CCR is a commercial feature released in open source Elasticsearch of the Platinum edition. After you purchase an Alibaba Cloud Elasticsearch cluster and make a few simple configurations, you can use this feature free of charge.

You can use CCR in disaster recovery scenarios to back up data among Elasticsearch clusters that reside in the same virtual private cloud (VPC) but different zones. If a cluster (remote cluster) fails, you can retrieve its data from another cluster (local cluster) and restore the data to the remote cluster. This feature helps prevent data loss.

To use CCR, you must prepare two types of clusters: local clusters and remote clusters. Remote clusters provide source data, which is stored in leader indexes. Local clusters replicate the data and store it in follower indexes. You can also use CCR to back up large volumes of data at a time in real time. For more information, see Cross-cluster replication.

Limits

  • You cannot reindex, search for, or replicate data between an Elasticsearch cluster deployed in the original network architecture and an Elasticsearch cluster deployed in the new network architecture. If you want to reindex, search for, or replicate data between clusters, make sure that the clusters are deployed in the same network architecture.

    Note
    • The network architecture of Alibaba Cloud Elasticsearch was adjusted in October 2020. Elasticsearch clusters that are created before October 2020 are deployed in the original network architecture. Elasticsearch clusters that are created in October 2020 or later are deployed in the new network architecture.

    • The time when the network architecture in the China (Zhangjiakou) region and the regions outside China was adjusted is uncertain. If you want to perform the preceding operations between a cluster created before October 2020 and a cluster created in October 2020 or later in such a region, submit a ticket to contact Alibaba Cloud technical support to check whether the clusters are deployed in the same network architecture.

  • For Elasticsearch clusters that are deployed in the original network architecture, only single-zone Elasticsearch clusters of V6.7.0 or later support the CCR feature. For Elasticsearch clusters that are deployed in the new network architecture, both single-zone and multi-zone Elasticsearch clusters of V6.7.0 or later support the CCR feature.

  • The CCR feature cannot be used to back up data between an Alibaba Cloud Elasticsearch cluster and a self-managed Elasticsearch cluster.

Procedure

  1. Make preparations

    Prepare a local cluster, a remote cluster, and a leader index.

  2. Step 1: Connect clusters

    Connect the remote cluster to the local cluster.

  3. Step 2: Add the remote cluster

    In the Kibana console of the local cluster, add the remote cluster.

  4. Step 3: Configure CCR

    In the Kibana console of the local cluster, configure the leader index and a follower index.

  5. Step 4: Verify the data backup result

    Insert data into the remote cluster and check whether the data is backed up in the local cluster.

Make preparations

  1. Create a local cluster and a remote cluster.

    For more information, see Create an Alibaba Cloud Elasticsearch cluster. The two clusters must be of the same version (V6.7.0 or later) and deployed in the same VPC and vSwitch.

    Important

    If you have uploaded a synonym file to the remote cluster, you must upload the synonym file to the local cluster.

  2. Log on to the Kibana console of the remote cluster. In the left-side navigation pane, click Dev Tools.

    For more information about how to log on to the Kibana console, see Log on to the Kibana console.

    Note

    In this example, an Elasticsearch V6.7.0 cluster is used as the remote cluster. Operations on clusters of other versions may differ. The actual operations in the console prevail.

  3. On the Console tab of the page that appears, run the following command to create a leader index.

    Important
    • If you create an index in an Elasticsearch cluster of V7.0 or earlier, you must enable the soft_deletes attribute for the index. Otherwise, an error is reported. You can run the GET /<yourIndexName>/_settings?pretty command to check whether the soft_deletes attribute is enabled. If the soft_deletes attribute is enabled, you can view the configuration of the soft_deletes attribute in the command output.

    • If you want to back up data in an existing index, you can call the reindex API to enable the soft_deletes attribute.

    PUT myindex
    {
      "settings": {
        "index.soft_deletes.retention.operations": 1024,
        "index.soft_deletes.enabled": true
      }
    }
  4. Disable the physical replication feature for the leader index.

    The physical replication feature is automatically enabled for indexes in Elasticsearch V6.7.0 clusters. Before you use CCR, you must disable the physical replication feature.

    1. Disable the index.

      POST myindex/_close
    2. Update the settings configuration of the index to disable the physical replication feature.

      PUT myindex/_settings
      {
      "index.replication.type" : null
      }
    3. Enable the index.

      POST myindex/_open

Step 1: Connect clusters

Configure the remote cluster to connect it to the local cluster. For more information, see Connect Elasticsearch clusters to enable cross-cluster searches. If the two clusters are connected, the information shown in the following figure appears.配置实例互通

Step 2: Add the remote cluster

  1. Log on to the Kibana console of the local cluster.

  2. In the left-side navigation pane, click Management.

  3. In the Elasticsearch section, click Remote Clusters.

  4. Click Add a remote cluster.

  5. In the Add remote cluster section, configure the parameters.

    Add remote cluster

    • Name: the name of the remote cluster. The name must be unique.

    • Seed nodes: the nodes in the remote cluster. Specify each node in the format of Node IP address:9300. To obtain the IP addresses of nodes, log on to the Kibana console of the remote cluster and run the GET /_cat/nodes?v command on the Console tab of the Dev Tools page. The nodes you specify must include a dedicated master node of the remote cluster. We recommend that you specify multiple nodes. This ensures that you can still use CCR if the specified dedicated master node fails.获取主节点IP地址

      Note

      During CCR, Kibana uses the IP addresses of data nodes to access clusters over TCP port 9300. HTTP port 9200 is not supported.

  6. Click Save.

    Then, the system connects the remote cluster to the local cluster. If the remote cluster is connected to the local cluster, Connected appears.连接成功

Step 3: Configure CCR

  1. Log on to the Kibana console of the local cluster. In the left-side navigation pane, click Management. In the Elasticsearch section of the page that appears, click Cross Cluster Replication.

  2. On the Follower indices tab, click Create a follower index.

  3. In the Add follower index section, configure the parameters.

    配置跨集群复制

    Parameter

    Description

    Remote cluster

    Select the cluster you added in Step 2: Add the remote cluster.

    Leader index

    The index whose data you want to back up. In this example, the myindex index that is created in Make preparations is used.

    Follower index

    The index to which you want to back up data. You must specify a unique index name.

  4. Click Create.

    After the follower index is created, the index is in the Active state.索引状态

Step 4: Verify the data backup result

  1. In the Kibana console of the remote cluster, run the following command to insert data into the remote cluster:

    POST myindex/_doc/
    {
      "name":"Jack",
      "age":40
    }
  2. In the Kibana console of the local cluster, run the following command to check whether the inserted data is backed up to the local cluster:

    GET myindex_follow/_search

    If the command is successfully run, the result shown in the following figure is returned.数据同步结果

    The preceding figure shows that data in the leader index myindex of the remote cluster is backed up to the follower index myindex_follow of the local cluster.

    Note

    The follower index myindex_follow is read-only. If you want to write data to the follower index, convert the follower index into a common index first. For more information, see Use Elasticsearch CCR to migrate data across data centers.

  3. Insert a document into the remote cluster and check whether the document is backed up to the local cluster in real time.

    POST myindex/_doc/
    {
      "name":"Pony",
      "age":50
    }

    Query the inserted document in the local cluster. The following figure shows the document.验证数据同步的实时性

    The preceding figure shows that the CCR feature can implement real-time backup of incremental data.

    Note

    You can also call the APIs for the CCR feature to perform cross-cluster replication operations. For more information, see Cross-cluster replication APIs.

FAQ

  • Q: I can use port 9300 to add a remote cluster. Why is only port 9200 accessible when I use a domain name to access an Elasticsearch cluster?

    A: Port 9300 is an open port. However, when you access a cluster over the Internet, Server Load Balancer (SLB) enables only port 9200 during port verification for security purposes.

  • Q: How do I view the status of CCR-based data synchronization?

    A: Run the GET /_ccr/stats command in the Kibana console of the cluster that stores the follower index and view the value of the number_of_failed_follow_indices parameter. This parameter indicates the number of failed shards.

    • If the value of the parameter is 0, the synchronization is normal.

    • If the value of the parameter is not 0, run the following commands for the cluster that stores the follower index to pause and then resume the synchronization:

      POST /<follower_index>/_ccr/pause_follow
      POST /<follower_index>/_ccr/resume_follow