OpenSearch:Integrating with RDS/PolarDB instances outside the service region

Background

• Some Alibaba Cloud customers have ApsaraDB RDS resources in regions where OpenSearch is not available.

• You need to search for documents.

• OpenSearch can connect only to RDS/PolarDB instances that are in the same region.

• Customers who have self-managed MySQL databases want to connect to OpenSearch.

Solution selection

• Requirements:

  • Connect OpenSearch to RDS/PolarDB instances that are in different regions.

  • Forward data from instances in different regions to the same region as the OpenSearch instance.

  • Ensure that data forwarding is lossless, ordered, and maintains the data structure.

  • Support full and incremental data import into OpenSearch.

• Solutions:

  • (Non-real-time) RDS/PolarDB -> MaxCompute -> OpenSearch

  • (Near-real-time) RDS/PolarDB (different region) -> DTS -> RDS/PolarDB (same region) -> OpenSearch

  • (Near-real-time) RDS/PolarDB (different region) -> DTS (full) -> RDS/PolarDB (same region) -> OpenSearch + API push

• Comparison and evaluation:

Data source	Full	Incremental	Resource consumption	Implementation difficulty	Potential costs
DTS	Help and support	Support	RDS + DTS (data migration - full + incremental)	Easy	RDS billing and daily DTS bills
MaxCompute	Support	Supported (daily scheduled)	MaxCompute Professional Edition (monthly) + dedicated cluster resources + Cloud Enterprise Network + resource authorization + region restrictions	Hard	MaxCompute resources and dedicated clusters
API/SDK + DTS	Support	Support	Development and debugging costs / DTS (data migration - full)	Medium	RDS billing + DTS (data migration - full)

Implementation steps

1. Create an RDS/PolarDB instance in the destination region.

   1. Create the target instance in the target region.

   2. Create a database and a management account that has Data Definition Language (DDL) and Data Manipulation Language (DML) permissions.

2. Create and configure a DTS task.

   1. Purchase Data Transmission Service (DTS). You may need to activate the service first.

   2. In the region of the destination OpenSearch instance, configure a data migration task.

   3. For more information about how to configure the DTS task, see the Data migration section in the DTS documentation.

      1. If your task involves a cross-region instance migration, the source RDS instance must have a public endpoint. You can request a public endpoint in the RDS console.

      2. You do not need to modify the IP address whitelist for data security purposes. The connection check passes even if the whitelist contains only 127.0.0.1.

3. Create an OpenSearch instance.

   1. After the full data migration to the destination RDS/PolarDB instance is complete, you can configure the OpenSearch application.

   2. For more information about how to configure an RDS/PolarDB data source for the application, see the Data Source section in the OpenSearch documentation.

   3. Data latency depends on two data forwarding steps: the DTS migration and the OpenSearch import. The overall latency is minimal. For more information about OpenSearch, see the Help Center.