Dataphin:Create a real-time custom source type

Limits

Support is limited to open-source Flink (Apache Flink) and Ververica Flink compute engines.

Customize Ververica Flink Connector instructions

After finalizing the customization of real-time source types in Dataphin, proceed to the Ververica Flink real-time computing management console to register the connector. For detailed instructions, see Upload and use custom connectors.

Should you need to update the connector code, it is necessary to apply the updates on both the Dataphin platform and the Ververica Flink real-time computing platform. The connector Jar file uploaded to the Ververica Flink platform will be the one utilized.

Procedure

1. Navigate to the Dataphin home page and single click the top menu bar Management Center -> Datasource Management.

2. To access the Create Offline Custom Source Type page, follow the steps below:

   Single click Custom Source Type -> Create Custom Source Type -> Create Real-time Data Source Type.

   

3. On the Create data source type page, fill in the required parameters as follows:

   Parameter	Description
   Type name	Enter a custom data source type name, limited to 128 characters.
   Type encoding	Specify the encoding for the custom data source type, adhering to the following rules: Must be globally unique. Cannot exceed 64 characters. Accepts only English letters, numbers, and underscores (_). Must not begin with a number. Note Once set, it cannot be changed.
   JAR package	Upload the JAR package for the data source type connector. File extension must be .jar (case insensitive). The file size must not exceed 500MB. Must not conflict with existing or reserved data source connectors. Should only contain data source-level encryption parameters. Note For guidance on custom real-time connectors, refer to the Flink official website.
   Configuration file	Upload the connector's configuration file and click Download configuration file template for assistance. File extension must be .yaml (case insensitive). Must not conflict with existing or reserved data source connectors. Should only contain data source-level encryption parameters. The configuration file details are as follows: # Declare the usage type of the connector (source table, target table, dimension table) kind: source sink dim # Define the connector name, format is CUSTOM_([A-Z|0-9]+) connector: CUSTOM_SOCKET # hostname is a property of the connector, displayed in the DDL with parameters. # isSensitive: Set to false to indicate that the target property (e.g., hostname) will not be displayed encrypted on the page. # defaultValue: Specifies the default value of the target property. # isRequired: Set to true to indicate that this property is required and displayed with an asterisk (*) on the page. # module: When the value of module is datasource, it means that the target property can be configured on the data source page; when the value of module is table, it means that the target property can be configured on the real-time metadata table page. hostname: isSensitive: false defaultValue: localhost isRequired: true module: datasource port: isSensitive: false # format: Defines the serialization format of data input or output, and its module is usually configured as table. format: name: changelog-csv module: table format_property: line-delimiter: isSensitive: false defaultValue: '/n' isRequired: true column-delimiter: isSensitive: false defaultValue: ','
   Description	Provide a concise description of the real-time custom source type, not exceeding 128 characters.

4. Click Confirm to finalize the creation of the custom real-time data source type.

What to do next

Upon successful creation of the real-time custom source type, the system will generate configuration items for this data source type. You can then integrate this data source type into Dataphin. For further details, see Data source.