All Products
Search
Document Center

Realtime Compute for Apache Flink:Manage UDFs

Last Updated:May 12, 2023

This topic describes how to manage user-defined functions (UDFs) in fully managed Flink. For example, you can register, update, and delete a UDF.

Prerequisites

If another Alibaba Cloud account or a RAM user wants to use fully managed Flink, you have granted permissions to the Alibaba Cloud account or the RAM user to perform operations by using your Alibaba Cloud account. For example, the permission is granted to manage UDFs. For more information, see Grant permissions to an account.

Precautions

To prevent conflicts between JAR package dependencies, take note of the following items when you develop UDFs:

  • Make sure that the Flink version that you select on the Draft Editor page is the same as the Flink version that is specified in the POM dependency.

  • Specify <scope>provided</scope> for Flink-related dependencies.

  • Use the Shade plug-in to package other third-party dependencies. For more information, see Apache Maven Shade plug-in.

For more information about how to handle Flink dependency conflicts between JAR file, see How do I troubleshoot dependency conflicts of Flink?

Register a UDF

Before you can use a UDF in SQL statements, register the UDF. Only registered UDFs can be used in SQL statements.

  1. Log on to the Realtime Compute for Apache Flink console.

  2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.

  3. In the left-side navigation pane, click SQL Editor.

  4. On the left side of the SQL Editor page, click the UDFs tab.

  5. Click Register UDF Artifact.

  6. In the Register UDF Artifact dialog box, upload a UDF JAR file.Register UDF Artifact

    You can use one of the following methods to upload a UDF JAR file:

    • Upload a file: Click Click to select next to Select a file to upload the UDF Artifact file that you want to upload. If you want to upload a dependency file, click Click to select next to Dependencies to upload the file that your UDF Artifact file depends on.

      Note
      • Your UDF JAR file is uploaded and stored in the sql-artifacts directory of the Object Storage Service (OSS) bucket that you select. Fully managed Flink parses the UDF JAR file and checks whether the classes of the Flink user-defined scalar function (UDF), user-defined aggregate function (UDAF), and user-defined table-valued function (UDTF) interfaces are used in the file. Then, fully managed Flink automatically extracts the class names and specifies the class names in the Function Name field.

      • The dependencies of Java UDFs can be added to UDF JAR packages or uploaded by using a dependency file. We recommend that you upload the dependencies of Python UDFs by separately uploading the dependency file.

    • Use External URL: Enter an external URL. If the size of the JAR file exceeds 200 MB or you want to use a JAR file that is stored in another service, you can use the external URL to obtain the JAR file.

      Note
      • If the size of the JAR file exceeds 200 MB, you can save the file of the UDF in the sql-artifacts/namespaces/{namespace} directory of the OSS bucket that is associated with fully managed Flink. Then, use the HTTPS path of the file.

      • You can use the HTTPS path of another service. In this case, the service and Realtime Compute for Apache Flink must reside in the same virtual private cloud (VPC). You can also establish a network connection between the service and Realtime Compute for Apache Flink over the Internet and then use the public endpoint. For more information, see How does a fully managed Flink service access the Internet?

  7. Click Confirm.

  8. In the Available Functions section of the Manage Functions dialog box, select one or more UDFs that you want to register and click Create Functions.

    In the UDFs list on the left side of the SQL Editor page, you can view all the registered UDFs.

Update a UDF

If you add a UDF to a UDF JAR file or change the code of a registered UDF in the file, you can perform the following operations to update the UDF JAR file:

  1. Log on to the Realtime Compute for Apache Flink console.

  2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.

  3. In the left-side navigation pane, click SQL Editor.

  4. On the left side of the SQL Editor page, click the UDFs tab.

  5. In the UDFs list, move the pointer over the name of the UDF whose JAR file you want to update and click the Update a UDF JAR file icon.

  6. In the Register UDF Artifact dialog box, upload a UDF JAR file.Update a UDF JAR file

    You can use one of the following methods to upload a UDF JAR file:

    • Upload a file: Click Click to select next to Select a file to upload the UDF Artifact file that you want to upload. If you want to upload a dependency file, click Click to select next to Dependencies to upload the file that your UDF Artifact file depends on.

    • External URL: Enter an external URL.

    Important
    • The UDF JAR file that you upload must contain all the classes of the registered UDFs.

    • The code in the new UDF JAR file takes effect only when you restart the draft for the deployment or publish a new draft. The code in the new UDF JAR file does not take effect on deployments that are running. The deployments that are running continue to use the original UDF JAR file.

  7. Click Update.

Delete a UDF

If you no longer need a UDF JAR file, perform the following operations to delete the UDF JAR file:

  1. Log on to the Realtime Compute for Apache Flink console.

  2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.

  3. In the left-side navigation pane, click SQL Editor.

  4. On the left side of the SQL Editor page, click the UDFs tab.

  5. In the UDFs list, move the pointer over the name of the UDF JAR file that you want to delete and click the Delete a UDF JAR file icon.

    Note

    Before you delete a UDF JAR file, make sure that the UDF that is registered by using the UDF JAR file is not referenced by a deployment or an SQL file.

  6. Select Delete associated files.

    If you want to delete the UDF JAR file, you must delete all registered UDFs from the file to avoid dirty data.

  7. Click Confirm.