Data integration is a simple and efficient data synchronization platform built on Dataphin. It provides powerful data pre-processing and enables high-speed, stable data synchronization between disparate data sources.
Background information
With the increasing application of big data across industries, data integration must meet several demands. These include the simple and efficient configuration of sync tasks for numerous data tables, the integration of multiple disparate data sources, light pre-processing of data, and the optimization of data sync tasks with features such as fault tolerance, speed limiting, and concurrency.
Function overview
If you purchased Dataphin after April 2020, the data synchronization feature has been upgraded to data integration.
Dataphin has upgraded its data integration capabilities to help you build a simple, efficient, secure, and reliable data synchronization platform:
Enhances data integration efficiency by supporting full database migration. This feature lets you quickly generate batch sync tasks and create destination tables with one click. No manual table creation is required for data synced to MaxCompute. For more information, see Configure an integration task using full database migration.
The flow and transform components support data pre-processing operations such as data cleansing, transformation, field desensitization, calculation, merging, distribution, and filtering. For more information, see Create an integration task using a single pipeline.
Supports Dev-Prod and Basic development modes, allowing you to flexibly choose a development mode based on your needs.
Supports quick synchronization of logical tables created in Dataphin to a destination database.
Supports custom components for features not natively available in the system to meet various data synchronization needs. Components for Relational Database Management Systems (RDBMS) connect through Java Database Connectivity (JDBC). For non-RDBMS database components, you must upload a JAR package.
Data integration supports various component types. You can generate an offline single pipeline by dragging, configuring, and assembling components. Data integration also supports the quick generation of batch sync tasks. For full database migration, the supported source databases are MySQL, SQL Server, and Oracle, and the supported destination is MaxCompute. Data integration also lets you create custom component types that are not supported by the system to meet your data synchronization needs.
Data integration entry points
Quick entry (recommended)
On the Dataphin home page, click Data Import in the Product Usage Path to go to the data integration page.
Standard entry
On the Dataphin home page, choose Develop > Data Integration from the top menu bar.
