Realtime Compute for Apache Flink:Manage custom connectors

Precautions

• For Realtime Compute for Apache Flink to correctly detect and use your custom connector, you must develop the connector according to the standards defined by the Flink community. Therefore, you must specify the connector's meta file and declare its Factory class. For more information about how to develop custom connectors, see User-defined Sources & Sinks.

• You can upload a connector of a specific type only once.

• To avoid JAR package dependency conflicts, note the following:

  • Keep the Flink runtime image version consistent with the Flink version specified in the Project Object Model (POM) dependencies.

  • Do not upload JAR packages at the runtime layer. This means you must add <scope>provided</scope> to the dependencies.

  • Package other third-party dependencies using the Shade method. For more information about Shade packaging, see Apache Maven Shade Plugin.

Upload and use a custom connector

1. Go to the custom connector registration page.

   1. Log on to the Realtime Compute for Apache Flink console.

   2. In the Actions column of the target workspace, click Console.

   3. In the navigation pane on the left, click Connectors.

2. Register the custom connector.

   1. On the Connectors page, click Create Custom Connector.

   2. Upload the custom connector JAR file.

      You can upload the custom connector JAR file in one of the following ways:

      • Upload File: Click Select File and choose the target connector JAR file.

      • External URL: To use a JAR file from another service, you can use the external URL feature to retrieve the file. For example, https://ossbucket/artifacts/namespaces/flink-default/flink-jobs-1.0-SNAPSHOT.jar

        Note

        Only the following two types of external URLs are supported:

        • A URL for the OSS bucket that you selected when you created the Flink workspace. You can view the attached OSS bucket on the Workspace Details page in the Realtime Compute for Apache Flink console.

        • A URL for another external storage system that Realtime Compute for Apache Flink has permission to access (public-read or granted permission).

   3. After the upload is complete, click Next.

      The system parses the content of the custom connector that you uploaded. If the parsing is successful, you can proceed to the next step. If the parsing fails, confirm that the code of your custom connector complies with Flink community standards.

   4. Click Finish.

      The created custom connector appears in the connector list.

3. You can use the connector in the job DDL.

   For more information about job development, see Job development map.

   Note

   The value of the connector parameter in the WITH clause is the value of the identifier parameter of DynamicTableFactory in your custom connector's JAR package. Other WITH parameters and their meanings are determined by the custom connector that you developed.

Update a custom connector

1. Go to the custom connector update page.

   1. Log on to the Realtime Compute for Apache Flink console.

   2. In the Actions column of the target workspace, click Console.

   3. In the navigation pane on the left, click Connectors.

2. On the Custom Connectors tab, find the target custom connector and click Edit next to its name.

3. Upload the custom connector JAR file.

   You can upload the custom connector JAR file in one of the following ways:

   • Upload File: Click Select File and choose the target connector JAR file.

   • External URL: Enter an external URL. For example, https://ossbucket/artifacts/namespaces/flink-default/flink-jobs-1.0-SNAPSHOT.jar

     Note

     Only the following two types of external URLs are supported:

     • A URL for the OSS bucket that you selected when you created the Flink workspace. You can view the attached OSS bucket on the Workspace Details page in the Realtime Compute for Apache Flink console.

     • A URL for another external storage system that Realtime Compute for Apache Flink has permission to access (public-read or granted permission).

4. After the upload is complete, click Next.

   The system parses the content of the custom connector that you uploaded. If the parsing is successful, you can proceed to the next step. If the parsing fails, confirm that the code of your custom connector complies with Flink community standards.

5. Click Finish.

References

• Flink provides a variety of built-in connectors. For more information about supported built-in connectors, see Supported connectors.

• For more information about using community edition Change Data Capture (CDC) connectors, see Use community edition CDC connectors.

• Flink provides data management by allowing you to create a catalog to manage and access the associated metadata. For more information, see Data Management.