Dataphin:Create a Hudi data source

Create a Hudi data source to enable Dataphin to read from or write data to Hudi.

Background information

Apache Hudi is a big data storage system that incorporates core repository and database functions directly into the database, with record-level insert, update, and delete support. Apache Hudi official website.

Permissions

Only users with custom global roles that have the Create Data Source permission and users with the super administrator, data source administrator, domain architect, or project administrator role can create data sources.

Procedure

1. On the Dataphin homepage, click Management Center > Data Source Management in the top navigation bar.

2. On the Datasource page, click +Create Data Source.

3. In the Create Data Source page, select Hudi from the Big Data section.

   Alternatively, select Hudi from the Recently Used section or search for it by keyword.

4. On the Create Hudi Data Source page, configure the connection parameters.

   1. Configure basic information.

      Parameter	Description
      Data Source Name	The name must meet the following requirements: It can contain only Chinese characters, letters, digits, underscores (_), and hyphens (-). It cannot exceed 64 characters in length.
      Datasource Code	Use the data source code to reference tables in Flink_SQL tasks: data_source_code.table_name or data_source_code.schema.table_name format. To auto-resolve the data source by environment, use ${data_source_code}.table or ${data_source_code}.schema.table. For more information, see Dataphin data source table development method. Important The data source code cannot be modified after it is configured. You can preview data on the object details page in the asset directory and asset checklist only after the data source code is configured. In Flink SQL, only MySQL, Hologres, MaxCompute, Oracle, StarRocks, Hive, SelectDB, and GaussDB data warehouse service (DWS) data sources are currently supported.
      Data Source Description	A brief description of the data source. It cannot exceed 128 characters.
      Data Source Configuration	Select a configuration mode: Select Production + Development Data Source if you use separate production and development environments. Select Production Data Source if you use a single shared environment.

   2. Configure connection parameters.

      Configure connection information for both environments if you selected Production + Development Data Source, or only the production environment if you selected Production Data Source.

      Note

      We recommend separate data sources for production and development to isolate environments. Dataphin also supports using the same data source with identical parameters for both.

      Parameter	Description
      Storage Configuration	Supports HDFS or OSS storage.
      Storage Path	HDFS Storage: Enter the HDFS storage path. Ensure the Flink user has access. Format: hdfs://host:port/path. OSS Storage: Enter the OSS storage path. Example: oss://dp-oss/hudi/. OSS storage also requires the Endpoint, AccessKeyID, and AccessKeySecret. Endpoint: Enter the endpoint for your OSS region. See OSS Region and Endpoint reference table for public cloud. AccessKeyID, AccessKeySecret: The AccessKey pair of the OSS account owner. See Obtain an AccessKey pair.
      Metadata Synchronization	Synchronizes Hudi table schemas to the Hive MetaStore. For HDFS storage with metadata synchronization enabled, configure: Version: Supports CDH6:2.1.1 and CDP7.1.3:3.1.300. Synchronization Mode: Select hms or jdbc. Each mode requires different parameters: hms: Thrift address of the Hive metastore and the target database name. Important hms mode requires the Hive metastore server to be running. jdbc: JDBC address, username, and password of the Hive metastore database, and the target database name. For OSS storage with metadata synchronization enabled, configure: Synchronization Mode: Default is hms and cannot be modified. Metadata Target Database: Default is DLF and cannot be modified. DLF Service Region Name: The region domain name of the DLF service. See DLF Region and Endpoint reference table. DLF Service Endpoint: The endpoint address of the DLF service. See DLF Region and Endpoint reference table. Database Name to Synchronize to Hive: Enter the database name to synchronize to Hive.