DataHub:Overview

Data synchronization of DataHub

DataHub provides data source and sink features, also known as data synchronization. You can use this feature to synchronize data from DataHub topics to other Alibaba Cloud services in real time or near real time. This process enables data to flow between different Alibaba Cloud services. DataHub supports data synchronization to MaxCompute (formerly ODPS), AnalyticDB for MySQL (ADS), ApsaraDB RDS, Tablestore, Object Storage Service (OSS), Elasticsearch, and Function Compute. You can write data to DataHub once, configure data synchronization, and then use the data in any of the supported cloud services to create a complete, closed-loop data solution.

Supported Alibaba Cloud services

Note:

• For MaxCompute, data is synchronized every 5 minutes or when a batch reaches 64 MB.

• For Tablestore, OSS, Elasticsearch, Function Compute, or Hologres, data is synchronized every 20 seconds or when a batch reaches 4 MB.

• For MySQL, ApsaraDB RDS, or AnalyticDB for MySQL 3.0, data is synchronized every 20 seconds or when a batch reaches 512 KB.

Usage notes

1. When you synchronize data, use Alibaba Cloud services that are in the same region. This practice helps avoid network connectivity issues and reduces network latency between regions.

2. Due to network isolation in Alibaba Cloud, you must use the internal endpoint (classic network) of the destination service during configuration. Synchronization over the public internet is not supported.

3. DataHub data synchronization supports only at-least-once semantics. In rare cases, such as a network error, this may result in a small amount of duplicate data at the destination service. You must handle data deduplication at the destination.

4. When you create a sync task, DataHub automatically attaches a SubscriptionId to the task. This ID records the read point in DataHub, not the synchronization point. Do not modify this SubscriptionId.

5. Only the owner or creator of a DataHub topic has the permission to create a connector (sync task).

IP whitelists

Service whitelist

MaxCompute lets you configure a project-level IP whitelist. If you configure an IP whitelist, DataHub cannot synchronize data to MaxCompute. To resolve this issue, MaxCompute provides a service whitelist feature. You must add the DataHub service name to the MaxCompute service whitelist to allow DataHub to synchronize data to MaxCompute.

The service name for DataHub is DataHub. You can set the service whitelist using the software development kit (SDK) or odpscmd.

To set the whitelist using the SDK, see the following example:

public static void setServiceName() throws OdpsException {
    String endpoint = "";
    String project = "";
    String accessId = "";
    String accessKey = "";

    Account account = new AliyunAccount(accessId, accessKey);
    Odps odps = new Odps(account);
    odps.setEndpoint(endpoint);
    odps.setDefaultProject(project);

    Map<String, String> prop = new HashMap<>();
    // Add the DataHub service name to the MaxCompute service whitelist. To add multiple services, separate them with commas. Each time you set this property, it overwrites the previous setting.
    prop.put("odps.security.ip.whitelist.services", "DataHub");
    Group group = new Groups(odps).get("groupName");
    group.updateProject(project, null, null, prop);

    // Check if the current MaxCompute service whitelist is set as expected.
    prop = odps.projects().get(project).getAllProperties();
    System.out.println("current service list: " + prop.get("odps.security.ip.whitelist.services"));
}

To set the whitelist using odpscmd, see the following example:

After you start odpscmd, run the following command to apply the configuration.

setproject odps.security.ip.whitelist.services=DataHub;

After the configuration is complete, you can verify that the configuration was successful.

setproject;
## If the output contains the odps.security.ip.whitelist.services=DataHub configuration item, the setting was successful.