Documentation

Configure a PolarDB for MySQL data source

Updated at:
Copy as MD
Product

This topic describes how to set up the network and account permissions for data synchronization.

Prerequisites

Before you configure the data source, complete the following preparations:

  • Source and destination data sources: You have purchased a PolarDB for MySQL cluster to use as the source data source and a destination data source. The destination data source can be MaxCompute, Hologres, Elasticsearch, DataHub, or Kafka. This topic uses an Alibaba Cloud PolarDB for MySQL cluster as the source data source in this example.

  • Data integration resources: You have purchased and configured an exclusive resource group for data integration. For more information, see Plan and configure resources.

  • Network plan: Before you perform data integration, you must establish a network connection between your data sources and the exclusive resource group for data integration based on your business requirements. After network connectivity is established, refer to this topic to configure access settings such as vSwitches and whitelists.

    • If the data source and the exclusive resource group for data integration are in the same region and the same virtual private cloud (VPC), they can connect to each other by default.

    • If the data source and the exclusive resource group for data integration are in different network environments, you must connect them by using a VPN gateway or another method.

Background information

When you synchronize data from a source data source to a destination data source, you must ensure network connectivity and the required account permissions between the data source and the DataWorks exclusive resource group for data integration.

  • Network whitelist

    This topic uses an example where the data source and resource group are in the same VPC. You must add the vSwitch CIDR block of the exclusive resource group to the whitelist of the data source to ensure the resource group can access the data source.实时同步单表数据（PolarDB)

  • Account permissions

    You must use a database account that has permission to access the data source. This account is used for data extraction and writing during data integration.

  • Other access restrictions

    If you use PolarDB for MySQL as the source data source, you must enable binary logging. PolarDB for MySQL is a cloud-native database that is fully compatible with MySQL. By default, it uses advanced physical logs instead of binary logs. To better integrate with the MySQL ecosystem, PolarDB for MySQL allows you to enable this feature.

Limits

  • Data synchronization solutions support only PolarDB for MySQL data sources, not other types of PolarDB.

  • For PolarDB for MySQL, real-time synchronization can be performed only from the primary node (read/write instance).

  • XA ROLLBACK is not supported. For transaction data prepared with an XA PREPARE statement, real-time synchronization writes the data to the destination. If an XA ROLLBACK statement is later executed, the rollback operation is not synchronized to the destination. To handle a scenario that involves XA ROLLBACK, you must manually remove the tables affected by XA ROLLBACK from the real-time synchronization task, add the tables again, and then perform a new full data initialization and incremental data synchronization.

Procedure

  1. Configure a whitelist.

    Add the vSwitch CIDR block of the exclusive resource group to the whitelist of your PolarDB for MySQL cluster. The steps are as follows:

    1. View and record the network information of the exclusive resource group for data integration.

      1. Log on to the DataWorks console.

      2. In the left-side navigation pane, click Resource Groups.

      3. On the Exclusive Resource Group tab, find the target exclusive resource group for data integration and click View Information.

      4. Copy the EIP address and Network segment from the dialog box to the database whitelist. In the details panel that appears, view the EIP address and CIDR block information.

      5. On the Exclusive Resource Group tab, click Network Settings for the target Data Integration resource group.

      6. On the VPC Binding tab, view the vSwitch CIDR Block to add to the database whitelist.

    2. Add the recorded elastic IP address and vSwitch CIDR block to the whitelist of the PolarDB for MySQL cluster.

      In the left-side navigation pane of the PolarDB console, choose Configuration and Management > Cluster Whitelist. Then, click Add IP Whitelist Group to add the elastic IP address and the vSwitch CIDR block. For more information, see Set a whitelist.

  2. Create an account and grant permissions.

    You must create a database account that has the SELECT, REPLICATION SLAVE, REPLICATION CLIENT permissions.

    1. Create a database account.

      For more information, see Create a database account.

    2. Grant permissions.

      You can run the following command to grant the required permissions to the account. Alternatively, you can grant the SUPER permission to the account.

      -- CREATE USER 'sync_account'@'%' IDENTIFIED BY 'password';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'sync_account'@'%';

  3. Enable binary logging for the PolarDB for MySQL cluster.

    For more information, see Enable binary logging.

Next steps

After you complete these configurations, your data sources and exclusive resource group for data integration are connected without access restrictions. You can then add the data sources to DataWorks and use them in a data synchronization task. For more information about how to add a data source, see Add a data source.