Data source config

Last Updated: Mar 23, 2017

Data source configuration is the primary task of data integration. During data synchronization (data import or export) task development, the project administrator should configure reachable data sources to support the entire data development project.

The project administrator can create, edit, and delete data sources in the current project. Currently multiple data source types are supported, including: ODPS, OCS, DRDS, ADS, RDS (MySQL, SQL Server, PostgreSQL), OSS, Oracle, and FTP.

New data source

[Scenario 1] Create an ODPS data source

The project administrator can follow the steps below to create an ODPS data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Project Management in the top menu bar and navigate to the Manage Data Sources page.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select ODPS as the data source type.

Step 5: Configure the information items of the ODPS data source.

1

The configuration items of the ODPS data source are described as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently.

ODPS Endpoint: Read-only by default. The setting is automatically read from the system configuration.

ODPS Project Name: Identifier of the corresponding ODPS project.

Connect as Logged in User: Connect to the ODPS project with the current user identity. If this option is selected, “Access Id” and “Access Key” options will be hidden and require no configuration.

Connect as System Administrator: Connect to the data source with the ODPS Project Owner identity.

  • AccessID: The AccessID corresponding to the ODPS Project Owner cloud account.
  • AccessKey: The AccessKey corresponding to the ODPS Project Owner cloud account. It is paired with the AccessID.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Description] A default data source (odps_first) will be generated for each project, and the ODPS project name is also the computing engine ODPS project name referenced by the current project.

[Scenario 2] Create a MySQL data source

The project administrator can follow the steps below to create a MySQL data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Project Management in the top menu bar and navigate to the Manage Data Sources page.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select RDSMySQL as the data source type.

Step 5: Select to configure the MySQL data source in the form of a RDS instance or JDBC.

Select to configure the MySQL data source in the form of a RDS instance:

1

Specific descriptions of the configuration items in the figure above are as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently (RDS>MySQL>RDS instance form).

RDS Instance ID: The ID of the MySQL data source instance.

RDS Instance Purchaser ID: The purchaser ID of the MySQL data source instance.

[Note] If you have selected the JDBC format to configure the data source, the format of the JDBC connection information is: jdbc:mysql://IP:Port/database.

Database Name: The database name of the data source.

User Name/Password: The user name and password of the database.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 3] Create an SQL Server data source

The project administrator can follow the steps below to create an SQL Server data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select RDSSQL Server as the data source type.

Step 5: Select to configure the SQL Server data source in the form of a RDS instance or JDBC.

Select to configure the SQL Server data source in the form of a RDS instance:

1

Specific descriptions of the configuration items in the figure above are as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently (RDS>SQL Server>RDS instance form).

RDS Instance ID: The ID of the SQL Server data source RDS instance.

RDS Instance Purchaser ID: The purchaser ID of the data source RDS instance.

[Note] If you have selected the JDBC format to configure the data source, the format of the JDBC connection information is: jdbc:mysql://IP:Port/database.

Database Name: The database name of the data source.

User Name/Password: The user name and password of the database.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 4] Create a PostgreSQL data source

The project administrator can follow the steps below to create a PostgreSQL data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select RDSPostgreSQL as the data source type.

Step 5: Select to configure the PostgreSQL data source in the form of a RDS instance or JDBC.

Select to configure the PostgreSQL data source in the form of a RDS instance:

1

Specific descriptions of the configuration items in the figure above are as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently (RDS>PostgreSQL>RDS instance form).

RDS Instance ID: The ID of the SQL Server data source RDS instance.

RDS Instance Purchaser ID: The purchaser ID of the data source RDS instance.

[Note] If you have selected the JDBC format to configure the data source, the format of the JDBC connection information is: jdbc:mysql://IP:Port/database.

Database Name: The database name of the data source.

User Name/Password: The user name and password of the database.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 5] Create an Oracle data source

The project administrator can follow the steps below to create an Oracle data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select Oracle as the data source type.

Step 5: Configure the information items of the Oracle data source.

1

The configuration items of the Oracle data source are described as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently.

Network Type: The current network type selected.

JDBCUrl: The JDBC URL. Format:

jdbc:oracle:thin:@serverIP:Port:Database.

User name/password: The user name and password.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 6] Create an ADS data source

The project administrator can follow the steps below to create an ADS data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select ADS as the data source type.

Step 5: Configure the information items of the ADS data source.

1

The configuration items of the ADS data source are described as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently.

ADSUrl: The ADS URL. Format: serverIP:Port.

Schema: The ADS Schema information.

AccessID/AceessKey: The user name and password, that is, the AccessKey pair.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 7] Create an OSS data source

The project administrator can follow the steps below to create an OSS data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select OSS as the data source type.

Step 5: Configure the information items of the OSS data source.

1

The configuration items of the OSS data source are described as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently.

Network Type: The current network type selected.

Endpoint: OSS endpoint information. Format: http://oss.aliyuncs.com.

Bucket: The OSS bucket information.

AccessID/AceessKey: The user name and password, that is, the AccessKey pair.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 8] Create an OCS data source

The project administrator can follow the steps below to create an OCS data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select OCS as the data source type.

Step 5: Configure the information items of the OCS data source.

1

The configuration items of the OCS data source are described as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently.

Network Type: The current network type selected.

PROXY: The OCS Proxy.

Port: The OCS port.

User Name/Password: The user name and password.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 9] Create a DRDS data source

The project administrator can follow the steps below to create a DRDS data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select DRDS as the data source type.

Step 5: Configure the information items of the DRDS data source.

1

The configuration items of the DRDS data source are described as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently.

Network Type: The current network type selected.

JDBCUrl: The JDBC URL. Format: jdbc://mysql://serverIP:Port/database.

User Name/Password: The user name and password.

Step 6: Click Test Connectivity after you have completed the configuration of the information items above.

Step 7: Click OK when the connectivity test is passed.

[Scenario 10] Create an FTP data source

The project administrator can follow the steps below to create an FTP data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Click New Data Source.

Step 4: In the New Data Source pop-up box, select FTP as the data source type.

Step 5: Configure the information items of the FTP data source.

1

The configuration items of the FTP data source are described as follows:

Data Source Name: A data source name may consist of letters, numbers, and underscores. It must begin with a letter or an underscore and cannot exceed 30 characters in length.

Data Source Descriptions: A brief description of the data source. The description should not exceed 1,024 characters in length.

Data Source Type: The data source type selected currently.

Network Type: The current network type selected.

Protocol: Currently only FTP and SFTP are supported.

Host: The FTP host IP address.

Port: If FTP is selected, the port is 21 by default; if SFTP is selected, the port is 22 by default.

User Name/Password: The account and password for accessing the FTP service.

Edit data sources

The project administrator can follow the steps below to change the configuration information of an existing data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Enter the data source name in the search box of the data source list to make a fuzzy match search of the data source to be edited.

Step 4: Click Edit in the action bar.

1

Step 5: Configure the data source information items (for details, see “Section 5.1.2.2 New Data Source).

Step 6: Click Test Connectivity after you have completed editing the information items.

Step 7: Click OK when the connectivity test is passed.

Delete data sources

The project administrator can follow the steps below to delete a data source:

Step 1: Go to Alibaba Cloud Dataplus platform > Data IDE Kit > Console as a developer, click the Enter Work Zone in the action bar of the corresponding project.

Step 2: Click Manage Projects in the top menu bar, and then click Manage Data Sources in the left navigation bar.

Step 3: Enter the data source name in the search box of the data source list to make a fuzzy match search of the data source to be deleted.

Step 4: Click Delete in the action bar.

1

[Attention] The Project administrators should exercise caution in editing and deleting a data source to avoid production faults resulting from interrupted normal execution of the workflows and code that reference the data source.

Thank you! We've received your feedback.