All Products
Search

This Product

    Dataphin:Manage data sources

    Document Center

    Dataphin:Manage data sources

    Last Updated:Mar 05, 2025

    Before utilizing Dataphin, you can connect the necessary data sources for integration. These sources enable the reading of raw data and writing back to the sources. Dataphin supports various data source types, including data warehouses like MaxCompute and Hive, along with traditional databases such as MySQL and Oracle.

    Permission description

    Only custom global roles with the Create Data Source Permission Point and roles like super administrator, data source administrator, module architect, and project administrator are authorized to create data sources.

    Feature introduction

    • Multiple Data Source Types Access: Dataphin offers a comprehensive selection of built-in data source types. These include big data storage, file, message queue, relational, and NoSQL data sources. For more information, see Dataphin supported data sources.

    • Custom Data Source Types: If the necessary data source type is not available among the built-in options, you can customize offline or real-time data source types. These custom data sources can then be connected to Dataphin to fulfill various access requirements. For specific operations, see:

    • Production and Development Data Sources: When adding data sources in Dataphin, you can specify them as production or development sources. The Prod environment for Basic and Dev-Prod projects interacts with production data sources, while the Dev environment for Dev-Prod projects engages with development data sources.

    • Data Source Tags: You can categorize data sources by business or type using custom tags for effective management.

      Note

      Sync tasks do not support dual environments for production and development. Data sources in sync tasks are treated as production data sources.

    Data source management entry

    On the Dataphin home page, click the top menu bar Management Center > Datasource Management .

    View data source list

    Once a data source is created, you can view, edit, and delete it from the data source list.

    image

    Area

    Description

    ①Function area

    • My responsibility: After checking, it will filter the data sources for which the current logged-in account is responsible,

    • Search: You can enter the name of the data source for fuzzy search.

    • Filter: Click image.png to filter data sources based on data source type, data source tag, and owner.

    • Manage tags: Dataphin supports tag management of data sources to support you in managing data sources by business scenarios. To create, see Manage data source tags.

    • Create A New Data Source: If you need to create a new data source. For specific operations, see Dataphin supported data sources.

    • Refresh: Click image.png to refresh the current data source.

    ②Data source card

    Dataphin displays the created data sources in a card format for you. This includes details such as name, type, application scenarios, tags, owner, description, update time, and connection information. You can also perform operations such as view application scenarios, view creation user/creation time, view connection information, test connection, Edit, Change Owner, Delete.

    • View application scenarios: Hover the mouse over the image application scenario icon after the data source type to view the application scenarios supported by the data source.

    • Edit: You can click the Edit button in the upper right corner of the development or production data source to modify the data source description and connection information. You can also click the Edit button in the upper right corner of a single data source card to modify the entire data source's name, tag, or configure data source encoding.

      Note

      • After the data source encoding is successfully configured, it cannot be modified.

      • After the data source encoding is successfully configured, data preview can be performed on the object details page of the asset directory and asset checklist.

    • Change Owner: In the Change Owner dialog box, select Owner, and click Confirm to transfer the data source owner to another Dataphin member.

    • Delete: You can click the Delete button in the upper right corner of the development or production data source to delete the data source in the development or production environment. You can also click the Delete button in the upper right corner of a single data source card to delete the entire data source. Once deleted, the data source cannot be revoked. Please proceed with caution.

      Important

      • When deleting a data source, the system will automatically check the reference status of the current data source in Dataphin. If there are referenced objects, it is recommended to delete the objects that depend on this data source first. If you forcefully delete, please proceed with caution. Once deleted, it cannot be revoked.

      • When deleting a data source, the system will not check the dependencies of SQL queries and Notebook files in the analysis platform. It is recommended to check whether there are dependencies in the SQL queries and Notebook files in the analysis platform before performing the delete operation.

    • View creation user/creation time: Hover the mouse over the image view icon after the update time to view the creator and creation time of the data source.

    • View connection information: You can click Connection followed by View to view connection information or perform a copy.

    • Test connection: You can click Connection followed by Test to test the network connectivity in the Test Network Connectivity dialog box between the default cluster or custom cluster and the data source.

    ③Batch operation area

    You can click Enable Batch Delete Mode to select the data sources to be deleted for batch deletion.