Elasticsearch:Cross-region Elasticsearch data replication

Step 1: Connect cross-region VPCs with CEN

Use Cloud Enterprise Network (CEN) to establish a cross-region connection between the VPCs of the leader and follower clusters. For detailed steps, see Connect VPCs across different regions.

Step 2: Obtain the leader cluster's private IP

1. Log on to the Alibaba Cloud Elasticsearch console.

2. On the Basic Information page of the leader cluster, find the Internal Endpoint field and copy the private domain name.

3. From an ECS instance in the same VPC, run the following command to resolve the private IP address of the leader cluster:

   ping <Private domain name of the leader cluster>

   Record the resolved IP address for later use.

Step 3: Create an NLB and a server group

Create an NLB instance in the VPC of the follower cluster to forward traffic to the leader cluster.

1. Log on to the Network Load Balancer (NLB) console.

2. Create a server group.

   1. In the left-side navigation pane, click ServerGroup.

   2. Click Create Server Group and configure the following parameters:

      Parameter	Description
      Server Group Type	Select IP type.
      Forwarding Port	Enable Multi-port Forwarding.
      Health Check	Set the port to 9300.

   3. In the server group that you created, click Add Backend Server. Add the private IP address of the leader cluster that you obtained in Step 2. Keep the default port.

3. Create an NLB instance and a listener.

   The NLB purchase page does not support creating a listener with the full-port feature. Therefore, first create an NLB instance without a listener. Then, on the instance details page, create a listener and enable the full-port feature.

   If you already have an NLB instance, you can create a listener directly. If not, you must create an NLB instance first.

   1. In the left-side navigation pane, click Cluster, and then select or create an NLB instance.

   2. Go to the instance details page, click the Listener tab, and then click Create Listener.

   3. Enable the Full-port feature and set the listener port range to 9200-9300.

   4. For ServerGroup, select the IP type and the server group that you created in the previous step.

Step 4: Configure PrivateLink

Use PrivateLink to establish a network connection from the managed VPC of the follower cluster to the leader cluster.

1. Log on to the PrivateLink console.

2. Create an endpoint service.

   1. In the left-side navigation pane, click Endpoint Service.

   2. Click Create Endpoint Service and configure the following parameters:

      Parameter	Description
      Service resource type	Select NLB.
      Service Resource	Select the NLB instance that you created or used in Step 3.
      Zone	Select the availability zone where the NLB instance is located.
      Automatically Accept Endpoint Connections	Select Yes.

3. Add a private connection in the follower cluster.

   1. Log on to the Alibaba Cloud Elasticsearch console and go to the instance details page of the follower cluster.

   2. In the left-side navigation pane, choose Configuration and Management > Security Settings.

   3. In the Network Settings section, click Configure Private Connection.

   4. Click Add Private Connection and select the endpoint service that you created in the previous step.

   5. Wait until the connection status changes to Connected.

4. Obtain the PrivateLink domain name.

   After the connection is established, return to the PrivateLink console. On the Endpoint Connection Status tab of the endpoint service, view the automatically created endpoint connection. This is the PrivateLink domain name that you will use to configure the remote cluster.

Step 5: Configure Cross-Cluster Replication (CCR)

1. Log on to Kibana for the follower cluster. For detailed steps, see Log on to the Kibana console over the internet (v2/v3 deployment architecture).

   On the instance details page of the follower cluster, click Data Visualization in the left-side navigation pane, and then click Go to Kibana.

2. Add a remote cluster.

   1. In the left-side navigation pane of Kibana, click Stack Management.

   2. In the Data section, click Remote Clusters.

   3. Click Add a remote cluster and configure the following parameters:

      Parameter	Description
      name	Enter the instance ID of the leader cluster.
      Proxy mode	Enable proxy mode.
      Proxy address	Enter the PrivateLink endpoint domain name that you obtained when you configured the private connection for the follower cluster, in the format <domain-name>:9300. The domain name is displayed on the Security page of the follower cluster instance, under Configure Private Connection.

   4. Click save and confirm that the connection status is Connected.

3. Configure a CCR replication mode.

   CCR supports two modes:

   Mode	Description
   Follower indices	Replicates a single, specified index.
   Auto-follow patterns	Automatically replicates indexes that match a specified name pattern. This mode is suitable for bulk synchronization. After creating an auto-follow pattern, new indexes created on the leader cluster are automatically synchronized to the follower cluster. Existing indexes are not synchronized automatically. To synchronize an existing index, you must manually create a follower index on the follower cluster.

   The following example uses an auto-follow pattern:

   1. In Stack Management, click Cross-Cluster Replication.

   2. Select the Auto-follow patterns tab and click Create an auto-follow pattern.

   3. For Remote cluster, select the remote cluster you added in the previous step. For Index patterns, enter * to replicate all indexes. To replicate only specific indexes, enter an index name pattern, such as logs-*.

   4. Click Create.

Step 6: Verify data synchronization

Verify synchronization for existing indices

Indices that existed in the leader cluster before you created the auto-follow pattern are not replicated automatically. You must manually create follower indices in Kibana of the follower cluster by navigating to Stack Management > Cross-Cluster Replication > Follower Indices.

After creating the follower index, run the following command in Kibana of the follower cluster to verify that the data is synchronized:

GET /<index_name>/_search

If the returned result is consistent with the result from the leader cluster, the existing data has been synchronized.

Verify synchronization for new data

1. In Kibana of the leader cluster, create a new index and write a document to it:

   PUT /test-increment-index
   
   POST /test-increment-index/_doc
   {
       "title": "increment test",
       "content": "This is a test document for CCR incremental sync."
   }

2. In Kibana of the follower cluster, query the index:

   GET /test-increment-index/_search

   If the returned document matches the source document, real-time synchronization is successful.