Enable Public Network Access for UDF & Spark Jobs - MaxCompute - Alibaba Cloud - MaxCompute

You can use this method to access a public IP address or domain name using a MaxCompute user-defined function (UDF), Spark, MapReduce (MR), PyODPS, or Mars.

Usage notes

Supported top-level domain names: aliyun.com, amap.com, dingtalk.com, alicloudapi.com, cainiao.com, alicdn.com, taobao.com, alibaba.com, alipaydev.com, and alibabadns.com.
IPv6 addresses cannot be configured. The number of public IP addresses is not limited.
If an external network address fails automatic verification, delete the address and resubmit it. If the address is still required, you can submit a request form to configure the external network address. For more information, see Network activation process.

Procedure

Step 1: Prepare an account and a project

Before you enable network connectivity between MaxCompute and the target service, make sure that the following conditions are met.

Create a MaxCompute project. In a data lakehouse scenario, set the data type for the MaxCompute project to a Hive-compatible type.
To access a target service in a VPC, make sure that the VPC owner account, the Alibaba Cloud account used to access the MaxCompute project, and the administrator account of the target service environment or cluster belong to the same root account.

Step 2: Edit external network addresses in Project Management

For frequently used public IP addresses or domain names, such as aliyun.com, you can directly add and delete them in the Project Management section of the MaxCompute console:

Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose Manage Configurations > Projects.
On the Projects page, click Manage in the Actions column for the target project.
On the Project Settings page, click the Parameter Configuration tab.
In the MaxCompute External Network section, click Edit.
Set the external network addresses that MaxCompute is allowed to access.
Click Submit.

For public IP addresses or domain names that fail automatic verification, perform the following steps:

Submit a request form to add the public IP addresses or domain names and ports to the whitelist.
After the MaxCompute technical support team receives your request, they review it and complete the configuration. The request is typically processed within three business days. After the request is processed, you can proceed with the subsequent steps. If you have any objections to the review result, you can submit a ticket.

Step 3: Access the public network address

When you use a SQL UDF, Spark, or MaxFrame job to access the public network, add the following configurations.

For other types of jobs, adjust the configuration information based on the job type.

SQL user-defined function (UDF) jobs

The parameters are set as follows:

-- Set the public IP address or domain name and port that you configured in the network connectivity request form. This is the public IP address or domain name that the following SQL statement will access.
-- To access multiple domain names or ports, separate them with commas (,).
SET odps.internet.access.list=<ip_address:port|realm_name:port>;
-- Execute the SQL statement to call the UDF.
SELECT <UDF_name>("<http://ip_address|realm_name>");

ip_address:port | domain_name:port: Required. The target public IP address or domain name and port.
UDF_name: The name of the UDF used to access the public IP address or domain name.

The following code provides an example of a job:

package com.aliyun.odps.test.udf;
import com.aliyun.odps.udf.UDF;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
public class <UDF_name> extends UDF {
    public String evaluate(String urlStr) throws IOException {
        URL url = new URL(urlStr);
        StringBuilder sb = new StringBuilder();
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()))) {
            String line;
            while ((line = reader.readLine()) != null) {
                sb.append(line).append('\n');
            }
        }
        return sb.toString();
    }
}

Execution: Assume that the name of the UDF created based on the sample code is url_fetch. After the network connectivity request is approved, you can run the following sample commands.
```
SET odps.internet.access.list=www.aliyun.com:80;
SELECT url_fetch("http://www.aliyun.com");
```

Spark on MaxCompute jobs

Add this configuration item to the conf file of the Spark client or to the configuration items when you submit a Spark job in DataWorks.

spark.hadoop.odps.cupid.smartnat.enable = true;
spark.hadoop.odps.cupid.internet.access.list=<ip_address:port>

MaxFrame jobs

The parameters are set as follows:

from maxframe import options
options.sql.settings = {
    "odps.internet.access.list": "<host>:80,<host>:443",
}

(Optional) Step 4: Add IP addresses to the whitelist

If access control is enabled on the server side, add the egress IP addresses that MaxCompute uses for public network access to the service whitelist.