MaxCompute:Network enablement process

Network structure

The following figure shows the network structure between MaxCompute and target services.

Network enablement solutions

1. MaxCompute Internet access solution

   • Scenarios

     This solution applies when you use a MaxCompute user-defined function (UDF), Spark, MapReduce (MR), or PyODPS/Mars to access a target IP address or domain name on the Internet.

   • Operation overview

     • For public IP addresses or domain names, such as aliyun.com, you can directly add or delete them in the Project Management section of the MaxCompute console.

     • For public IP addresses or domain names that fail automatic validation, you must submit a ticket. The review is completed within three business days. If the request is approved, you can access the target IP address or domain name, provided it has no security restrictions.

     • If the target public IP address or domain name has access control restrictions, contact the service administrator to remove them. This ensures that MaxCompute can access the public IP address or domain name.

2. MaxCompute VPC access solution (dedicated connection)

   • Scenarios

     This solution applies when you use MaxCompute SQL, a UDF, Spark, PyODPS/Mars, a foreign table, or a data lakehouse architecture to access services in a VPC, such as RDS, HBase clusters, and Hadoop clusters.

   • Operation overview

     • Log on to the VPC console to grant permissions to MaxCompute. Then, create a connection between MaxCompute and the VPC in the MaxCompute console.

     • After the connection is created, you can view the elastic network interface (ENI) created by MaxCompute in the MaxCompute console. The access rules for this ENI are controlled by the security group that you provided when you created the VPC connection.

     • If the target service has access control enabled, add the ENI IP address or the vSwitch CIDR block to the service's whitelist.

     • When MaxCompute accesses a VPC, it establishes a network connection only to the specified VPC ID. To access other VPCs in the same or different regions, you can use other VPC solutions to connect them to the VPC that is configured in the dedicated connection.

3. Accessing specific Alibaba Cloud services

   • Scenarios

     This solution applies when you use MaxCompute SQL, a UDF, Spark, PyODPS/Mars, a foreign table, or a data lakehouse architecture to access Alibaba Cloud services such as OSS, DLF, Tablestore, and Hologres. The connection uses the internal network addresses provided by Alibaba Cloud services.

   • Operation overview

     • If you create an OSS or Tablestore foreign table, you can access the table only through the internal endpoint of OSS or Tablestore.

     • If you call a UDF to access OSS or Tablestore, you can access them only through their public endpoints.

     • To access Hologres using a foreign table, you must use a classic network domain name. For more information, see the Classic Network address in Endpoints.

4. Accessing MaxCompute over a VPC PrivateLink

5. VPC peering connection solution

   This solution applies when you use a VPC in one region to access a MaxCompute service in another region.

6. Using a CEN solution to interconnect VPCs

   This solution applies when you use Cloud Enterprise Network (CEN) to interconnect VPCs across regions.

High concurrency leads to DNS resolution failure

Problem description: During the execution of a UDF or Spark task, many concurrent requests are generated to access the target domain name. This causes DNS resolution to fail.

Solution: Resolve the domain name to an IP address during the task initialization phase. Then, use the resolved IP address for access during the execution phase. For more information, see High Concurrency Causes DNS Resolution Failures.

Accessing HTTPS services using an IP address

Problem description: When a Spark or UDF task accesses a remote service in a VPC, such as KMS or OSS, it must use HTTPS. If you directly access the target service using an IP address, an error is reported.

Solution: Add the domain name to the host of the request. This resolves the validation error that occurs when you directly access an HTTPS service using an IP address. For more information, see Use an IP address to access an HTTPS service.