Before using Dataphin, choose a database or data warehouse as your data source. This source is used to read raw data and write data during development. Dataphin integrates many compute engines. It supports data warehouses such as MaxCompute and Hive, and also connects to traditional enterprise databases such as MySQL and Oracle.
Background information
Dataphin supports various data sources, including big data storage, file, message queue, relational, and NoSQL data sources. The data source types supported by each module are described below.
To connect to a data source in Dataphin, first create the data source in Data Source Management.
Dataphin lets you add both production and development data sources. Basic projects and the production (Prod) environment of Dev-Prod projects use production data sources. The development (Dev) environment of Dev-Prod projects uses development data sources. In DataService Studio, the Basic mode and the Prod environment of Dev-Prod mode read from production data sources, while the Dev environment reads from development data sources. Sync tasks do not support separate Dev and Prod environments and always use production data sources.
NoteIf a built-in data source type does not meet your needs, create a custom offline or real-time data source. You can then connect it to Dataphin. For more information, see:
Data source descriptions
Scenario | Description | References |
Offline integration | Offline integration supports various widgets, including input, output, and transform widgets. Create a single offline integration pipeline by dragging, configuring, and assembling these widgets on the canvas. Offline integration also provides a code editor for more customization. When you create a custom RDBMS data source, its input and output components are automatically added to the component library to meet diverse data synchronization needs. | |
Real-time integration | Dataphin's real-time integration feature synchronizes data changes from a source to a destination in real time. You can integrate changes from an entire database or all tables to keep the source and destination data sources synchronized. | |
Offline development - Database SQL | After you connect a data source to Dataphin, you can create Database SQL nodes in Dataphin for development. | |
Metadata acquisition | The Metadata Center extracts, processes, centrally stores, and manages metadata from various business systems. This supports data governance and enhances the ability to organize, retrieve, and analyze data within your organization. | |
Real-time development | Connected data sources can be used to create real-time metatables and develop real-time nodes. | |
Data Quality | Asset Quality is a comprehensive data quality solution on the Dataphin platform for data development and use. You can create global table quality rules or data source quality rules based on your data sources. Data source quality rules let you create quality monitoring rules for any data source in Dataphin. All supported data sources can be tested for connectivity. However, only some data sources support quality rules for monitoring table schema changes. For more information, see the Data Quality - Table Schema Change column in the tables below. | |
DataService Studio | DataService Studio (OneService) is the final step in building a data mid-end using Dataphin. As a unified data service endpoint, DataService Studio provides centralized, marketplace-style data management. It lowers the barrier to data accessibility while ensuring data security. | |
Tag Factory | Tag Factory provides a one-stop platform for the entire lifecycle of a tag, from creation to service. It is designed for data R&D teams and developers and is suitable for scenarios like risk control and marketing. The platform supports the development, management, and exploration of offline, real-time, and service tags. This empowers business applications and helps you build valuable tag assets that are efficient, discoverable, usable, and manageable. |
This topic lists the data sources that can be connected to Dataphin and their supported application scenarios. For details about the specific features supported by each data source in different scenarios, see:
Big data storage data sources
Data source type | Offline integration | Real-time integration | Offline development - Database SQL | Metadata acquisition | Real-time development | Global table quality | Data Quality - Table Schema Change | DataService Studio | Tag Factory | Creation guide |
MaxCompute | Supported | Supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Supported | |
Hive | Supported | Supported | Not supported | Supported | Supported | Supported | Supported | Not supported | Not supported | |
Hologres | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | |
Impala | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Supported | Not supported | |
TDH Inceptor | Supported | Not supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Not supported | |
Kudu | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | |
StarRocks | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
Hudi | Not supported | Not supported | Not supported | Not supported | Support | Not supported | Not supported | Not supported | Not supported | |
Doris | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
GreenPlum | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Supported | |
TDengine | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | |
ArgoDB | Supported | Not supported | Not supported | Not supported | Not supported | Supported | Supported | Not supported | Not supported | |
Paimon | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
SelectDB | Supported | Not supported | Supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
Lindorm (compute engine) | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | |
Databricks | Supported | Supported | Not supported | Not supported | Not supported | Supported | Supported | Supported | Not supported | |
Amazon Redshift | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
DolphinDB | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | Not supported | |
Snowflake | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported |
File data sources
Data source type | Offline integration | Real-time integration | Offline development - Database SQL | Metadata acquisition | Real-time development | Global table quality | Data Quality - Table Schema Change | DataService Studio | Tag Factory | Creation guide |
HDFS | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
FTP | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
OSS | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
Amazon S3 | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported |
Message queue data sources
Data source type | Offline integration | Real-time integration | Offline development - Database SQL | Metadata acquisition | Real-time development | Global table quality | Data Quality - Table Schema Change | DataService Studio | Tag Factory | Creation guide |
Log Service | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
Kafka | Supported | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | |
DataHub | Supported | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | |
RabbitMQ | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported |
Relational data sources
Data source type | Offline integration | Real-time integration | Offline development - Database SQL | Metadata acquisition | Real-time development | Global table quality | Data Quality - Table Schema Change | DataService Studio | Tag Factory | Creation guide |
PolarDB | Supported | Not supported | Not supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
PolarDB-X (formerly DRDS) | Supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Not supported | Not supported | |
PolarDB-X 2.0 | Supported | Not supported | Supported | Not supported | Not supported | Supported | Supported | Supported | Not supported | |
MySQL | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | |
SAP HANA | Supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
Microsoft SQL Server | Supported | Supported | Support | Supported | Supported | Supported | Supported | Supported | Not supported | |
PostgreSQL | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | |
AnalyticDB for MySQL 2.0 | Supported | Not supported | Supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | |
AnalyticDB for MySQL 3.0 | Supported | Not supported | Supported | Supported | Supported | Not supported | Supported | Supported | Not supported | |
AnalyticDB for PostgreSQL | Supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | |
OceanBase | Supported | Not supported | Supported | Supported | Supported | Not supported | Supported | Supported | Not supported | |
Oracle | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | |
Vertica | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | |
IBM DB2 | Supported | Supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | |
Teradata | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | |
ClickHouse | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
DM | Supported | Not supported | Supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
GBase 8a | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
KingbaseES | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
TiDB | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
GoldenDB | Supported | Not supported | Not supported | Not supported | Not supported | Supported | Supported | Not supported | Not supported | |
OpenGauss | Supported | Not supported | Supported | Supported | Not supported | Not supported | Not supported | Not supported | Supported | |
GaussDB (DWS) | Supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Supported | Not supported | |
Amazon RDS for MySQL | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
Amazon RDS for PostgreSQL | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
Amazon RDS for SQL Server | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
Amazon RDS for Oracle | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
Amazon RDS for DB2 | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | |
TDSQL for MySQL | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | |
GBase 8c | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported |
NoSQL data sources
Data source type | Offline integration | Real-time integration | Offline development - Database SQL | Metadata acquisition | Real-time development | Global table quality | Data Quality - Table Schema Change | DataService Studio | Tag Factory | Creation guide |
HBase0.9.4 | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Supported | Supported | |
HBase1.1x | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | |
HBase2.0 | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | |
Elasticsearch | Supported | Not supported | Not supported | Supported | Supported | Not supported | Not supported | Supported | supported | |
MongoDB | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | |
Tablestore | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | |
Aliyun HBase | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Supported | Not supported | Not supported | |
Redis | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
Lindorm (wide table) | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | |
Presto | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Easysearch | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Trino | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
OpenSearch | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported |
Semi-structured storage data sources
Data source type | Offline integration | Real-time integration | Offline development - Database SQL | Metadata acquisition | Real-time development | Global table quality | Data Quality - Table Schema Change | DataService Studio | Tag Factory | Creation guide |
API | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | |
SAP Table | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Salesforce | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Lark Bitable data source | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported |