Connect Elasticsearch clusters to enable cross-cluster searches - Elasticsearch

Prerequisites

The Elasticsearch clusters that you want to connect are created. For more information, see Create an Alibaba Cloud Elasticsearch cluster. The Elasticsearch clusters meet the following requirements:

They are of the same Elasticsearch version.
They belong to the same Alibaba Cloud account.
They reside in the same virtual private cloud (VPC).
They use the same deployment method.

Precautions

The network architecture of Alibaba Cloud Elasticsearch was adjusted in October 2020. Elasticsearch clusters created before October 2020 are deployed in the original network architecture. Elasticsearch clusters created in October 2020 or later are deployed in the new network architecture.You cannot perform cross-cluster operations, such as reindex, searches, or replication, between a cluster deployed in the original network architecture and a cluster deployed in the new network architecture. If you want to perform these operations between two clusters, you must make sure that the clusters are deployed in the same network architecture.

Note The time when the network architecture in the China (Zhangjiakou) region and the regions outside China was adjusted is uncertain. If you want to perform the preceding operations between a cluster created before October 2020 and a cluster created in October 2020 or later in such a region, submit a ticket to contact Alibaba Cloud technical support to check whether the clusters are deployed in the same network architecture.

Procedure

Step 1: Connect Elasticsearch clusters
Step 2: Configure the cross-cluster search feature

Step 1: Connect Elasticsearch clusters

Log on to the Elasticsearch console.
In the left-side navigation pane, click Elasticsearch Clusters.
Navigate to the desired cluster.
1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
2. In the left-side navigation pane, click Elasticsearch Clusters. On the Elasticsearch Clusters page, find the cluster and click its ID.
In the left-side navigation pane of the page that appears, choose Configuration and Management > Security.
In the Network Settings section, click Edit on the right side of Cluster Interconnection.
In the Edit Configuration panel, click Add Cluster.
In the Add Cluster dialog box, select the IDs of the remote clusters that you want to connect.
Notice
- The RAM users of an Alibaba Cloud account can query all clusters that belong to the account only after the ListInstance permission is granted to the RAM users. For more information, see Objects supported for authorization.
- After you connect the current cluster to a remote cluster, you can view the ID of the current cluster in the Edit Configuration panel of the remote cluster. This indicates that the communication between the two clusters is bidirectional.
Click OK.
After you add a cluster, you can find the cluster in the Connected Clusters section of the Edit Configuration panel.

Note If you no longer require the added cluster, click Remove in the Actions column to remove the cluster.

Step 2: Configure the cross-cluster search feature

After you connect clusters, you must configure the cross-cluster search feature. This way, you can search for data in the remote cluster from the current cluster.

In the remote cluster, prepare test data.

Note This step helps you verify the cross-cluster search feature. If your remote cluster already has test data, skip this step.
1. Log on to the Kibana console of the remote cluster and go to the homepage of the Kibana console as prompted.
  For more information about how to log on to the Kibana console, see Log on to the Kibana console.
  
  Note In this example, an Elasticsearch V6.7.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.
2. In the left-side navigation pane, click Dev Tools.
3. On the Console tab of the page that appears, run the related commands to create an index, add a document to the index, and insert data into the document in the remote cluster.
  Sample commands:
  1. Create an index
```
PUT /twitter
{
    "settings" : {
        "index" : {
            "number_of_shards" : 3, 
            "number_of_replicas" : 2 
        }
    }
}
```
  2. Create a document and insert data into the document
```
POST twitter/_doc/
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}
```
In the current cluster, configure the cross-cluster search feature and check whether the feature is correctly configured.
1. Log on to the Kibana console of the current cluster based on the instructions in Step 1. Then, use the IP addresses of the nodes in the remote cluster to configure the cross-cluster search feature for the current cluster.
  In this example, a V6.7 cluster is used. The methods that are used to configure the cross-cluster search feature for clusters of other versions are similar. For more information, see Cross-cluster search in Elasticsearch V7.X, Cross-cluster search in Elasticsearch V6.3, and Cross-cluster search in Elasticsearch V5.5.
  Sample command:
```
PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_one": {
          "seeds": [
            "10.8.xx.xx:9300",
            "10.8.xx.xx:9300",
            "10.8.xx.xx:9300"
          ]
        }
      }
    }
  }
}
```
  You can obtain the IP addresses of the nodes in the remote cluster from the Node Visualization tab of the Basic Information page for the cluster. For more information, see View the cluster status and node information.
  Notice
  
  Domain names cannot be used to configure the cross-cluster search feature.
  
  You can query the index data in the remote cluster from the current cluster. This applies only after you configure the node IP addresses of the remote cluster in the current cluster for cross-cluster searches. You cannot run similar commands in the remote cluster to search for data in the current cluster. To search for data in the current cluster from the remote cluster, you must add the node IP addresses of the current cluster to the remote cluster.
2. Run the following command to check whether the cross-cluster search feature is correctly configured:
```
POST /cluster_one:twitter/_doc/_search
{
  "query": {
    "match_all": {}
  }
}
```
  If the cross-cluster search feature is correctly configured, the following result is returned:
```
{
  "took" : 78,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "cluster_one:twitter",
        "_type" : "_doc",
        "_id" : "qudxxxxxxxxxx_7ie6J",
        "_score" : 1.0,
        "_source" : {
          "user" : "kimchy",
          "post_date" : "2009-11-15T14:12:12",
          "message" : "trying out Elasticsearch"
        }
      }
    ]
  }
}
```