edit-icon download-icon

Connectivity test of data sources

Last Updated: Sep 22, 2018
Data sourceData source typeNetwork typeConnectivity test supportAdd custom resource group
MySQL ApsaraDB Classic network Supported -
VPC Supported -
With public IP address Supported -
Without public IP address Unsupported Add custom resource groups
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
SQL Server ApsaraDB Classic network Supported -
VPC Supported -
With public IP address Supported -
Without public IP address Unsupported Add custom resource groups
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
PostgreSQL ApsaraDB Classic network Supported -
VPC Supported -
With public IP address Supported -
Without public IP address Unsupported Add custom resource groups
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
Oracle
With public IP address Supported -
Without public IP address Unsupported Add custom resource groups
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
DRDS ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups
HybridDB for MySQL ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups
HybridDB for PostgreSQL ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups
MaxCompute (for ODPS data sources) ApsaraDB Classic network Supported -
AnalyticDB (for ADS data sources) ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups
OSS ApsaraDB Classic network Supported -
VPC Supported -
HDFS
With public IP address Supported -
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
FTP
With public IP address Supported -
Without public IP address Unsupported Add custom resource groups
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
MongoDB ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups
With public IP address Supported -
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
Memcache ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups
Redis ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups
With public IP address Supported -
User-created ECS Classic network Supported -
VPC Unsupported Add custom resource groups
Table Store (for Table Store data sources) ApsaraDB Classic network Supported -
VPC In scheduling Add custom resource groups

Description

In the preceding table, “-“ indicates the item is unavailable, “No“ indicates that the connectivity test fails and a custom resource group must be added but the synchronization tasks can be configured.

Data sources in VPC environment:

  • Connectivity test for RDS data sources in VPC environment is supported.

  • Other data sources in VPC environment are under planning.

  • Connectivity tests are not supported for Financial Cloud networks.

User-created ECS data sources:

  • The classic network supports JDBC-based connectivity tests normally on the public network.

  • Currently, the VPC does not support connectivity tests.

  • Currently, connectivity tests for cross-region sources are not supported.

  • Connectivity tests for Financial Cloud networks are not supported.

Currently, data sync is implemented only by adding a custom resource group. For more information, see Data synchronization configuration for the VPC (Financial Cloud).

For user-created ECS data sources, make sure you add the IP address of the scheduling cluster to the security group for both inbound and outbound traffic (which is true for both the public network and classic network). If the security group is not added, it may disconnect during synchronization. For more information, see Add a security group. You cannot add an extensive range of ports on the ECS security group page. To add them, use the security group API of ECS. For more information, see AuthorizeSecurityGroup.

Data sources created in local IDCs or on the ECS server without public IP addresses:

Data sources created in local IDCs or on the ECS server with public IP addresses:

For such data sources, public-network-based JDBC is applied for connectivity tests. If the connectivity test fails, check the constraints of the local network or relevant databases.

Scheduling cluster description

  • Scheduling clusters are deployed in East China 2, South China 1, Hong Kong, and Singapore regions. Currently, as a rule of thumb, scheduling clusters in the region East China 2 are used for users’ data sources. Suppose that a user’s MongoDB data source is on the classic network of the North China region and scheduling clusters are on the classic network of the East China 2 region, cross-region communication is unavailable for both networks.

  • OXS clusters cannot communicate with ECS clusters on the intranet.

    The scheduling clusters of RDS are OXS clusters, which can communicate with RDS servers in any mainland China regions through the intranet. Scheduling clusters of other data sources are another set of scheduling clusters on the ECS classic network.

    For example, when you test the connectivity between the RDS server and the user-created database, data source connectivity tests for both the RDS server and the user-created database are successful. However, during the practical scheduling, the RDS server is assigned to an OXS scheduling cluster and the user-created database is assigned to an ECS scheduling cluster, resulting in to a connectivity failure test because the two clusters cannot communicate with each other. For most cases, we recommend that you change the connection mode of the RDS server to MySQL>JDBC so that both the RDS server and the user-created data can be assigned to the ECS cluster for intercommunication.

View the cluster for task running

  • If the data source type is RDS, the synchronization task runs in an OXS cluster. The logs are shown in the following figure:

    1

  • For other data source types, the synchronization tasks run in an ECS scheduling cluster, as shown in the following figure:

    1

  • If the scheduling cluster is a custom scheduling resource, the log is shown in the following figure (this is critical to determine if a custom resource group is used):

    1

  • Go to the Data Integration test page and click Run, you can find that all synchronization tasks are running in the ECS scheduling cluster. For this reason, users may report that RDS-related tasks can run manually but cannot be scheduled. This condition occurs because you must run those tasks in an OXS scheduling cluster when the data source type is RDS and cross-region communication is needed. In this condition, you must perform O&M Scheduling > Run Test.

Note:

For connectivity tests of data sources, you may concern the charge for public-network-based synchronization the most. The following example describes the data sync charge from RDS to MaxCompute:

Currently, Data Integration is free of charge. However, it may include a few charged products. No fee is incurred for configuring MaxCompute data sync in DataWorks. Charge is applied only when you want to manually add a parameter in the script mode to set a public IP address for the MaxCompute tunnel (however, this parameter is unavailable in the template generated in the script mode.)

Summary

  • When the connectivity test fails, any of the following conditions apply:

    • For the incorrect instance resulting from incorrect information, check whether the database name, user name, and password of the data source are correct:
  • For database disconnection, check the region where the data source resides, the network type, whether the whitelist is complete, and the instance ID:

    5.jpg | center

  • For network disconnection during synchronization:

    First, check the full log for the scheduling resource and whether the resource is a custom one.

    If so, check whether the IP address of the custom resource group has been added to the whitelist of the data source such as RDS (this also applies to MongoDB).

    Check whether the connectivity test between both data sources is succeeded and their whitelists are complete (if the whitelists are incomplete, the test result varies randomly; specifically, the test succeeds if the task is assigned to the added scheduling server and fails, if no scheduling server has been added.)

  • For the condition where the task is displayed as succeeded but the disconnection error 8000 can be found in the log:

    This condition occurs when the custom scheduling resource group is used and the IP address 10.116.134.123 and port 8000 are not set as permitted in the security group for inbound traffic. In this condition, add an IP address and a port, and run the task again.

Connectivity test failure example

Example 1

Problem

Test connection failed. Connectivity test of data source failed. An error occurred while connecting to the database. The database connection string is “jdbc:mysql://xx.xx.xx.x:xxxx/t_uoer_bradef”; the user name is “xxxx_test”; and the exception message is “Access denied for user “xxxx_test”@”%” to database “yyyy_demo””.

Solution

  1. Check whether the entered information is correct.

  2. Check whether the password, whitelist, or your account has the permission to access the database. You can add the required permissions in the RDS console.

Example 2

Problem

Test connection failed. Connectivity test of data source failed. The error message is as follows:

  1. error message: Timed out after 5000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=UNKNOWN, servers=[(xxxxxxxxxx), type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}]

Solution

For a non-VPC MongoDB, you must add whitelist for the connectivity test of the MongoDB data source. For more information, see Add a whitelist.

Thank you! We've received your feedback.