Elasticsearch:Replicate Elasticsearch data in the same region

Step 1: Connect VPCs by using CEN

Use Cloud Enterprise Network (CEN) to connect the VPC of the leader cluster and the VPC of the follower cluster. For detailed instructions, see Connect VPCs in the same region.

Step 2: Obtain the leader cluster's private IP

1. Log on to the Alibaba Cloud Elasticsearch console.

2. On the Basic Information page of the leader cluster, find the Internal Endpoint field and copy the private domain name.

3. From an ECS instance in the same VPC, run the following command to resolve the private IP address of the leader cluster:

   ping <Private domain name of the leader cluster>

   Record the resolved IP address for later use.

Step 3: Create an NLB and a server group

Create an NLB instance in the VPC of the follower cluster to forward traffic to the leader cluster.

1. Log on to the Network Load Balancer (NLB) console.

2. Create a server group.

   1. In the left-side navigation pane, click ServerGroup.

   2. Click Create Server Group and configure the following parameters:

      Parameter	Description
      Server Group Type	Select IP type.
      Forwarding Port	Enable Multi-port Forwarding.
      Health Check	Set the port to 9300.

   3. In the server group that you created, click Add Backend Server. Add the private IP address of the leader cluster that you obtained in Step 2. Keep the default port.

3. Create an NLB instance and a listener.

   The NLB purchase page does not support creating a listener with the full-port feature. Therefore, first create an NLB instance without a listener. Then, on the instance details page, create a listener and enable the full-port feature.

   If you already have an NLB instance, you can create a listener directly. If not, you must create an NLB instance first.

   1. In the left-side navigation pane, click Cluster, and then select or create an NLB instance.

   2. Go to the instance details page, click the Listener tab, and then click Create Listener.

   3. Enable the Full-port feature and set the listener port range to 9200-9300.

   4. For ServerGroup, select the IP type and the server group that you created in the previous step.

Step 4: Configure PrivateLink

Use PrivateLink to establish a network connection from the managed VPC of the follower cluster to the leader cluster.

1. Log on to the PrivateLink console.

2. Create an endpoint service.

   1. In the left-side navigation pane, click Endpoint Service.

   2. Click Create Endpoint Service and configure the following parameters:

      Parameter	Description
      Service resource type	Select NLB.
      Service Resource	Select the NLB instance that you created or used in Step 3.
      Zone	Select the availability zone where the NLB instance is located.
      Automatically Accept Endpoint Connections	Select Yes.

3. Add a private connection in the follower cluster.

   1. Log on to the Alibaba Cloud Elasticsearch console and go to the instance details page of the follower cluster.

   2. In the left-side navigation pane, choose Configuration and Management > Security Settings.

   3. In the Network Settings section, click Configure Private Connection.

   4. Click Add Private Connection and select the endpoint service that you created in the previous step.

   5. Wait until the connection status changes to Connected.

4. Obtain the PrivateLink domain name.

   After the connection is established, return to the PrivateLink console. On the Endpoint Connection Status tab of the endpoint service, view the automatically created endpoint connection. This is the PrivateLink domain name that you will use to configure the remote cluster.

Step 5: Configure cross-cluster replication (CCR)

1. Access Kibana for the follower cluster. For instructions, see Log on to the Kibana console over the internet (v2/v3 deployment architecture).

   On the instance details page of the follower cluster, click Data Visualization in the left-side navigation pane, and then click Access Kibana.

2. Add the remote cluster.

   1. In the left-side menu of Kibana, click Stack Management.

   2. In the Data section, click Remote Clusters.

   3. Click Add a remote cluster and configure the following parameters:

      Parameter	Description
      Name	Enter the instance ID of the leader cluster.
      Proxy mode	Enable proxy mode.
      Proxy address	Enter the PrivateLink domain name that you obtained in Step 4. The format is <Domain name>:9300.

   4. Click Save and confirm that the connection status is Connected.

3. Set up a replication method.

   CCR supports two methods:

   Mode	Description
   Follower index	Replicates a single, specified index.
   auto-follow pattern	Automatically replicates indices that match a specified index name pattern. This mode is suitable for bulk synchronization. An auto-follow pattern automatically replicates new indices created in the leader cluster but does not replicate existing ones. To replicate an existing index, you must manually create a corresponding follower index.

   The following example uses the auto-follow pattern:

   1. In Stack Management, click Cross-Cluster Replication.

   2. Click the Auto-follow patterns tab, and then click Create an auto-follow pattern.

   3. For Remote cluster, select the remote cluster you added in the previous step. For Index patterns, enter * to replicate all indices. If you only need to replicate specific indices, enter an index name pattern, such as logs-*.

   4. Click Create.

Step 6: Verify data synchronization

Verify synchronization for existing indices

Indices that existed in the leader cluster before you created the auto-follow pattern are not replicated automatically. You must manually create follower indices in Kibana of the follower cluster by navigating to Stack Management > Cross-Cluster Replication > Follower Indices.

After creating the follower index, run the following command in Kibana of the follower cluster to verify that the data is synchronized:

GET /<index_name>/_search

If the returned result is consistent with the result from the leader cluster, the existing data has been synchronized.

Verify synchronization for new data

1. In Kibana of the leader cluster, create a new index and write a document to it:

   PUT /test-increment-index
   
   POST /test-increment-index/_doc
   {
       "title": "increment test",
       "content": "This is a test document for CCR incremental sync."
   }

2. In Kibana of the follower cluster, query the index:

   GET /test-increment-index/_search

   If the returned document matches the source document, real-time synchronization is successful.