Dataphin supports the creation of custom offline data sources to accommodate various business scenarios for data synchronization.
Prerequisites
For RDBMS databases, retrieve the database driver files.
For other database types, retrieve the JSON code for datasource connection configuration items and read/write plug-ins.
Background information
RDBMS databases, such as MySQL, Oracle, and SQL Server, are relational databases. The connection method for these databases typically includes the database driver and configuration items, such as the connection address, username, and password for logon. Dataphin provides default datasource configuration items for RDBMS-type databases. You only need to prepare the database driver to create an RDBMS custom source type.
For non-RDBMS databases, the connection methods and configuration items for the datasource are more varied. You can define the necessary configuration items for datasource connection through JSON and upload the corresponding read/write plug-in driver packages to create non-RDBMS custom sources.
Integration component generation instructions
After creating an offline custom source type, the integration component for this type will be automatically generated in the integration pipeline component library. You can use this component for integration development. The instructions for generating integration components are as follows:
When creating an RDBMS-type offline custom datasource, input and output components are automatically generated in the open directory of the integration pipeline component library.
For other custom source types, the system generates the read/write plug-in components you uploaded in the open directory of the component library based on the uploaded read/write plug-ins.
Create a new RDBMS-type datasource
On the Dataphin home page, select Management Center > Datasource Management .
In the left-side navigation pane, select Datasource > Custom Source Type .
On the Custom Source Type page, click Create Custom Source Type , and select Create Offline Custom Source Type from the drop-down list.
On the Create Offline Custom Source Type page, configure the following parameters.
Parameter
Description
Basic Configuration
Type
Select RDBMS Database.
Name
The name of the custom component.
Supports Chinese, English letters in both uppercase and lowercase, underscores (_), and numbers. The name can be up to 64 characters in length.
Type Code
Used by the backend and cannot be edited after creation.
Supports only English letters in both uppercase and lowercase, numbers, and underscores (_), and cannot start with a number.
Resource Configuration
Driver Name
Enter the database driver name.
Supports English letters in both uppercase and lowercase, underscores (_), numbers, and half-width periods (.). The name can be up to 64 characters in length. For example, com.mysql.jdbc.Driver.
Driver Files
Uploaded database driver files.
NoteOnly .jar type files are supported, and the file size must not exceed 200MB.
Description
Description
A brief description of the custom datasource. The description can be up to 128 characters in length.
Click Create .
Create other types of datasources
On the Create Offline Custom Source Type page, configure the following parameters.
Parameter
Description
Basic Configuration
Type
Select Other Database.
Name
The name of the custom component.
Supports Chinese, English letters in both uppercase and lowercase, underscores (_), and numbers. The name can be up to 64 characters in length.
Type Code
Used by the backend and cannot be edited after creation.
Supports only English letters in both uppercase and lowercase, numbers, and underscores (_), and cannot start with a number.
Datasource JSON
Enter the datasource JSON code, which defines the configuration items of the datasource. The code example is explained as follows.
[ { "columnName": "url", "columnType": "NORMAL", "text": { "zh_CN": "链接地址", "en_US": "address", "zh_TW": "繁体" }, "placeholder": { "zh_CN": "请输入链接地址", "en_US": "input address", "zh_TW": "繁体" } }, { "columnName": "username", "columnType": "NORMAL", "text": { "zh_CN": "用户名", "en_US": "username", "zh_TW": "繁体" }, "placeholder": { "zh_CN": "请输入用户名", "en_US": "input username", "zh_TW": "繁体" } }, { "columnName": "password", "columnType": "ENCRYPT", "text": { "zh_CN": "密码", "en_US": "password", "zh_TW": "繁体" }, "placeholder": { "zh_CN": "请输入密码", "en_US": "input password", "zh_TW": "繁体" } } ]Resource Configuration
Read/write Plug-in
Select the type of plug-in to upload and fill in the ClassName (plug-in class name) and upload the plug-in.
Only .jar type files are supported, and the file size must not exceed 200MB.
At least one of the read plug-in or write plug-in must be selected for configuration.
If a read plug-in is selected, a read component will be generated. If a write plug-in is selected, a write component will be generated.
Description
Description
A brief description of the custom datasource, not exceeding 128 characters.
Click Create .
What to do next
After you have created the offline custom source type, the system will generate the datasource configuration items for this type. You can then integrate this type of datasource into Dataphin. For more information, see create a custom datasource.