Elasticsearch:Replicate Elasticsearch data in the same region

Step 1: Connect same-region VPCs with CEN

Use Cloud Enterprise Network (CEN) to connect the VPC of the leader cluster and the VPC of the follower cluster. For detailed instructions, see Connect VPCs in the same region.

Step 2: Obtain the leader cluster's private IP

1. Log on to the Alibaba Cloud Elasticsearch console.

2. On the Basic Information page of the leader cluster, find the Internal Endpoint field and copy the private domain name.

3. From an ECS instance in the same VPC, run the following command to resolve the private IP of the leader cluster:

   ping <Private domain name of the leader cluster>

   Record the resolved IP address for later use.

Step 3: Create an NLB instance and server group

Create an NLB instance in the VPC of the follower cluster to forward traffic to the leader cluster.

1. Log on to the Network Load Balancer (NLB) console.

2. Create a server group.

   1. In the left-side navigation pane, click ServerGroup.

   2. Click Create Server Group and configure the following parameters:

      Parameter	Description
      Server Group Type	Select IP type.
      Forwarding Port	Enable All-port forwarding.
      Health Check	Enter 9300 for the port.

   3. In the server group you created, click Add backend servers. Add the private IP of the leader cluster that you obtained in Step 2. Keep the default port setting.

3. Create an NLB instance and a listener.

   The NLB purchase page does not support the all-port feature during listener configuration. Create an NLB instance first without configuring a listener, and then enable the all-port feature when you create a listener on the instance details page.

   If you already have an NLB instance, you can create a listener directly. Otherwise, create an NLB instance first.

   1. In the left-side navigation pane, click Cluster, and then select or create an NLB instance.

   2. Go to the instance details page, click the Listener tab, and then click Create Listener.

   3. Enable the All-port feature and set the listener port range to 9200-9300.

   4. For ServerGroup, select the IP type and the server group that you created in the previous step.

Step 4: Configure PrivateLink

Use PrivateLink to connect the management VPC of the follower cluster to the network of the leader cluster.

1. Log on to the PrivateLink console.

2. Create an endpoint service.

   1. In the left-side navigation pane, click Endpoint Service.

   2. Click Create Endpoint Service and configure the following parameters:

      Parameter	Description
      Service resource type	Select NLB.
      Service Resource	Select the NLB instance created or used in Step 3.
      Zone	Select the availability zone where the NLB instance is located.
      Automatically Accept Endpoint Connections	Select Yes.

3. Add a private connection in the follower cluster.

   1. Log on to the Alibaba Cloud Elasticsearch console and go to the details page of the follower cluster instance.

   2. In the left-side navigation pane, choose Configuration and Management > Security Settings.

   3. In the Network Settings section, click Configure Private Connection.

   4. Click Add Private Connection and select the endpoint service that you created in the previous step.

   5. Wait until the connection status changes to Connected.

4. Obtain the PrivateLink domain name.

   After the connection is established, return to the PrivateLink console. On the Endpoint Connection Status tab of the endpoint service, find the automatically created endpoint connection. You will use this PrivateLink domain name to configure the remote cluster.

Step 5: Configure CCR

1. Log on to the Kibana console of the follower cluster. For alternative instructions on using a public endpoint, see Log on to the Kibana console of a v2/v3 cluster by using a public endpoint.

   On the instance details page of the follower cluster, click Data Visualization in the left-side navigation pane, and then click Go to Kibana.

2. Add a remote cluster.

   1. In the left-side navigation pane of Kibana, click Stack Management.

   2. In the Data section, click Remote clusters.

   3. Click Add a remote cluster and configure the following parameters:

      Parameter	Description
      Name	Enter the instance ID of the leader cluster.
      Proxy mode	Enable proxy mode.
      Proxy address	Enter the PrivateLink domain name obtained in Step 4 in the format <PrivateLink domain name>:9300.

   4. Click Save and make sure the connection status is connected.

3. Configure a CCR follow mode.

   CCR supports two modes:

   Mode	Description
   Follower indices	Replicates a single, specified index.
   Auto-follow pattern	Automatically replicates indices that match a specified index pattern. This mode is suitable for bulk synchronization. After you create an auto-follow pattern, new indices created in the leader cluster are automatically replicated to the follower cluster. Existing indices are not automatically replicated. To replicate an existing index, you must manually create a follower index in the follower cluster.

   The following steps use the auto-follow pattern as an example:

   1. In Stack Management, click Cross-Cluster Replication.

   2. Click the Auto-follow patterns tab, and then click Create an auto-follow pattern.

   3. For Remote cluster, select the remote cluster you just added. For Index patterns, enter * to replicate all indices. To replicate specific indices, you can enter a specific index pattern, such as logs-*.

   4. Click Create.

Step 6: Verify data synchronization

Verify synchronization for existing indices

If the leader cluster already contains indices before you create the auto-follow pattern, those indices are not automatically replicated. You must manually create follower indices in the follower cluster's Kibana console by navigating to Stack Management > Cross-cluster replication > Follower indices.

After you create a follower index, run the following command in the follower cluster's Kibana console to verify data synchronization:

GET /<index_name>/_search

If the returned result is consistent with the result from the leader cluster, the existing data is synchronized.

Verify synchronization for new data

1. In the leader cluster's Kibana console, create a new index and write a document to it:

   PUT /test-increment-index
   
   POST /test-increment-index/_doc
   {
       "title": "increment test",
       "content": "This is a test document for CCR incremental sync."
   }

2. In the follower cluster's Kibana console, query the index:

   GET /test-increment-index/_search

   If the returned document matches what you wrote, new data is being synchronized in real time.