All Products
Search
Document Center

Elasticsearch:Connect Elasticsearch clusters to enable cross-cluster searches

Last Updated:Oct 19, 2023

To ensure data security, Alibaba Cloud Elasticsearch clusters are isolated from each other by default. You can connect clusters to enable cross-cluster searches. This topic describes how to connect Elasticsearch clusters to enable cross-cluster searches.

Prerequisites

The Elasticsearch clusters that you want to connect are created and meet the following requirements. For more information about how to create an Elasticsearch cluster, see Create an Alibaba Cloud Elasticsearch cluster.

  • They belong to the same Alibaba Cloud account.

  • They are of the same Elasticsearch version.

  • They reside in the same virtual private cloud (VPC).

  • They use the same deployment method.

  • They are deployed in the same network architecture.

  • Note

    You cannot reindex, search for, or replicate data between an Elasticsearch cluster deployed in the original network architecture and an Elasticsearch cluster deployed in the new network architecture.

    • The network architecture of Alibaba Cloud Elasticsearch was adjusted in October 2020. Elasticsearch clusters created before October 2020 are deployed in the original network architecture. Elasticsearch clusters created in October 2020 or later are deployed in the new network architecture.

    • The time when the network architecture in the China (Zhangjiakou) region and the regions outside China was adjusted is uncertain. If you want to perform the preceding operations between a cluster created before October 2020 and a cluster created in October 2020 or later in such a region, submit a ticket to contact Alibaba Cloud technical support to check whether the clusters are deployed in the same network architecture.

Procedure

Step 1: Connect Elasticsearch clusters

  1. Log on to the Alibaba Cloud Elasticsearch console.
  2. In the left-side navigation pane, click Elasticsearch Clusters.
  3. Navigate to the desired cluster.
    1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
    2. On the Elasticsearch Clusters page, find the cluster and click its ID.
  4. In the left-side navigation pane of the page that appears, choose Configuration and Management > Security.

  5. In the Network Settings section, click Edit on the right side of Cluster Interconnection.

  6. In the Edit Configuration panel, click Add Cluster.

  7. In the Add Cluster dialog box, select the IDs of the remote clusters that you want to connect.

    添加实例页面
    Important
    • The RAM users of an Alibaba Cloud account can query all clusters that belong to the account only after the ListInstance permission is granted to the RAM users. For more information, see Elasticsearch objects supported for authorization.

    • After you connect the current cluster to a remote cluster, you can view the ID of the current cluster in the Edit Configuration panel of the remote cluster. This indicates that the communication between the two clusters is bidirectional.

  8. Click OK.

    After you add a cluster, you can find the cluster in the Connected Clusters section of the Edit Configuration panel.与当前实例打通的实例列表

    Note

    If you no longer require the added cluster, click Remove in the Actions column to remove the cluster.

Step 2: Configure the cross-cluster search feature

After you connect clusters, you must configure the cross-cluster search feature. This way, you can search for data in the remote cluster from the current cluster.

  1. In the remote cluster, prepare test data.

    Note

    This step helps you verify the cross-cluster search feature. If your remote cluster already has test data, skip this step.

    1. Log on to the Kibana console of the remote cluster and go to the homepage of the Kibana console as prompted.

      For more information about how to log on to the Kibana console, see Log on to the Kibana console.
      Note In this example, an Elasticsearch V6.7.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.
    2. In the left-side navigation pane of the page that appears, click Dev Tools.

    3. On the Console tab of the page that appears, run the related commands to create an index, add a document to the index, and insert data into the document in the remote cluster.

      Sample commands:

      1. Create an index.

        PUT /twitter
        {
            "settings" : {
                "index" : {
                    "number_of_shards" : 3, 
                    "number_of_replicas" : 2 
                }
            }
        }
      2. Create a document and insert data into the document.

        POST twitter/_doc/
        {
            "user" : "kimchy",
            "post_date" : "2009-11-15T14:12:12",
            "message" : "trying out Elasticsearch"
        }
  2. In the current cluster, configure the cross-cluster search feature and check whether the feature is correctly configured.

    1. Log on to the Kibana console of the current cluster based on the instructions in Step 1. Then, use the IP addresses of the nodes in the remote cluster to configure the cross-cluster search feature for the current cluster.

      In this example, a V6.7 cluster is used. The methods that are used to configure the cross-cluster search feature for clusters of other versions are similar. For more information, see Cross-cluster search in Elasticsearch V7.X, Cross-cluster search in Elasticsearch V6.3, and Cross-cluster search in Elasticsearch V5.5.

      Sample command:

      PUT _cluster/settings
      {
        "persistent": {
          "cluster": {
            "remote": {
              "cluster_one": {
                "seeds": [
                  "10.8.xx.xx:9300",
                  "10.8.xx.xx:9300",
                  "10.8.xx.xx:9300"
                ]
              }
            }
          }
        }
      }

      You can obtain the IP addresses of the nodes in the remote cluster from the Node Visualization tab of the Basic Information page for the cluster. For more information, see View the cluster status and node information.

      Important
      • Domain names cannot be used to configure the cross-cluster search feature.

      • You can query the index data in the remote cluster from the current cluster. This applies only after you configure the node IP addresses of the remote cluster in the current cluster for cross-cluster searches. You cannot run similar commands in the remote cluster to search for data in the current cluster. To search for data in the current cluster from the remote cluster, you must add the node IP addresses of the current cluster to the remote cluster.

    2. Run the following command to check whether the cross-cluster search feature is correctly configured:

      POST /cluster_one:twitter/_doc/_search
      {
        "query": {
          "match_all": {}
        }
      }

      If the cross-cluster search feature is correctly configured, the following result is returned:

      {
        "took" : 78,
        "timed_out" : false,
        "_shards" : {
          "total" : 3,
          "successful" : 3,
          "skipped" : 0,
          "failed" : 0
        },
        "_clusters" : {
          "total" : 1,
          "successful" : 1,
          "skipped" : 0
        },
        "hits" : {
          "total" : 1,
          "max_score" : 1.0,
          "hits" : [
            {
              "_index" : "cluster_one:twitter",
              "_type" : "_doc",
              "_id" : "qudxxxxxxxxxx_7ie6J",
              "_score" : 1.0,
              "_source" : {
                "user" : "kimchy",
                "post_date" : "2009-11-15T14:12:12",
                "message" : "trying out Elasticsearch"
              }
            }
          ]
        }
      }