This topic describes how MaxCompute can access services in a virtual private cloud (VPC), such as ApsaraDB RDS, HBase clusters, and Hadoop clusters. Supported access methods include MaxCompute SQL, UDFs, Spark, PyODPS/Mars, foreign tables, and data lakehouse architectures.
This solution creates two security groups for return traffic: MaxCompute-vpc-xxx and MaxCompute-backup-vpc-xxx, where xxx is the VPC ID that you provide. Do not modify the rules of these security groups or use them to manage security rules for other components. The platform is not responsible for any issues resulting from such modifications.
Limitations
Region and zone restrictions:
China (Hangzhou) - Zones H, I, J, and K;
China (Beijing) - Zones F, G, H, I, and L;
China (Shanghai) - Zones B, E, G, M, and N;
China (Zhangjiakou) - Zones A, B, and C;
China (Ulanqab) - Zones B and C;
China (Shenzhen) - Zones C, D, E, and F;
China (Hong Kong) - Zones B and C;
China East 2 (Shanghai) Finance - Zone F;
Japan (Tokyo) - Zones A and B;
Singapore - Zones A, B, and C;
Malaysia (Kuala Lumpur) - Zones A and B;
Indonesia (Jakarta) - Zones A and B;
Germany (Frankfurt) - Zones A, B, and C;
US (Silicon Valley) - Zones A and B;
US (Virginia) - Zones A and B.
Supported targets: VPC IP addresses or domain names, ApsaraDB RDS, HBase clusters, and Hadoop clusters.
Procedure
Step 1: Prepare account and project
Before you create a network connection between MaxCompute and the target service, make sure that the following prerequisites are met.
Create a MaxCompute project. For data lakehouse scenarios, we recommend that you set the data type of the project to Hive-compatible.
To access a service in a VPC, ensure that the Alibaba Cloud account that owns the VPC, the account used to access the MaxCompute project, and the administrator account for the target service all belong to the same primary Alibaba Cloud account.
Step 2: Create a direct network connection
1. Grant permissions
Authorize the operating user:
Only a Project Owner or a user with the tenant-level Super_Administrator or Admin role can perform this authorization. For more information, see Role planning.
For the authorization procedure, see List of object permissions in a tenant.
To allow MaxCompute to create elastic network interfaces (ENIs) in your VPC for connectivity, click Authorize while you are logged in to your Alibaba Cloud account.
2. Configure security group rules
In your VPC, create a dedicated security group to control MaxCompute's access to resources within the VPC.
We recommend that you create a new Basic Security Group. Avoid using other types of security groups or security groups that are already in use. MaxCompute will create an elastic network interface (ENI) in the VPC of the current user to access your services.
Configure the outbound rules of the security group to control which destination addresses MaxCompute jobs (via ENIs) can access. If you have no special requirements, you can leave the default outbound rules.
The inbound traffic to the ENI is return traffic. Therefore, you must allow all inbound traffic.
Log in to the Virtual Private Cloud console.
In the navigation pane on the left, choose VPC. In the top-left corner, select a region.
On the VPC page, click the Instance ID/Name of the target VPC.
On the VPC details page, click the Resource Management tab.
On the Resource Management tab, in the VPC Resources section, hover over the number next to Security Group and click Add.
For Security Group Type, select Basic Security Group.
A Basic Security Group allows outbound traffic by default. An Advanced Security Groups denies outbound traffic by default, which prevents access to any services in the VPC.
For Network, select the VPC that contains the service you want to access.
For more information about how to create a security group, see Create a security group.
Configure the security group to support the MaxCompute network.
In the Actions column of the target security group, click Manage Rules.
On the Access Rule tab, select the Inbound tab. In the Actions column of the target rule, click Edit. Configure the following parameters to allow all Inbound traffic.
Set Authorization policy to Allow.
Set Priority to 1.
Set Protocol to All.
For Source, add the CIDR block of the VPC or the CIDR block of the vSwitch where the service that you want to access is deployed.
Destination is ALL(-1/-1) by default.
For more information about how to configure rules, see Security group application guide and examples.
For HBase, if you cannot grant network permissions to a security group, you must add the IP address of the ENI that is created by MaxCompute to a whitelist. Because ENI IP addresses may change, we recommend that you add the IP address range of the vSwitch to which the VPC belongs. To obtain the ENI IP address, log on to the ECS console and click Network Interfaces in the left-side navigation pane.
During the network connection creation process, MaxCompute automatically creates two ENIs based on your bandwidth requirements. These ENIs are free of charge and are placed in this security group.
3. Create the network connection
An Alibaba Cloud account or a RAM user with the tenant-level Super_Administrator or Admin role for MaxCompute can create a connection between MaxCompute and a VPC in the MaxCompute console. For more information, see MaxCompute tenant-level roles. To create a connection:
Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose .
On the Network Connection page, click Add Network Connection.
In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.
In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.
The following table describes the parameters.
Parameter
Required
Description
Connection Name:
Required
A custom name for the connection. The name must meet the following requirements:
-
Start with a letter.
-
Contain only letters, underscores (_), and digits.
-
Be 1 to 63 characters in length.
Type:
Required
The default value is Passthrough.
A Passthrough connection is a VPC connection.
Region:
Required
The system automatically populates this parameter based on the region that you select in the upper-left corner. For more information, see Supported regions.
VPC Selected:
Required
A virtual private cloud (VPC) is an isolated virtual network. It provides a secure and configurable private network space, similar to a traditional data center.
-
The ID of the VPC. To create a new VPC, see Create or delete a VPC.
-
To obtain the ID:
-
Log in to the Virtual Private Cloud console.
-
In the navigation pane on the left, choose VPC. In the top-left corner, select a region.
-
On the VPC page, find the Instance ID/Name of the VPC.
If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.
-
Switch:
Required
A vSwitch is used to create a subnet. Different vSwitches in the same VPC can communicate with each other over the internal network. You can deploy cloud resources across vSwitches in different zones to protect your application from failures in a single zone.
-
The ID of the vSwitch that is attached to the VPC. If no vSwitches are available, see Create or delete a vSwitch.
-
To obtain the ID:
-
Log in to the Virtual Private Cloud console.
-
In the navigation pane on the left, choose vSwitch. In the top-left corner, select a region.
-
On the vSwitch page, find the Instance ID/Name of the vSwitch.
If you connect to an HBase or Hadoop cluster, you can find this information in the network connection details on the corresponding console.
-
Security group:
Required
A security group acts as a virtual firewall for your cloud resources. By managing security groups and their rules, you can implement fine-grained network isolation and access control.
The ID of the security group. To create a security group, see Create a security group.
-
4. Configure target service security group
After you complete the preceding operations to enable the ENI-based direct connection, you must add security rules to the target service. These rules authorize the security group representing MaxCompute to access specific service ports, such as 9200 and 31000.
For example, to access ApsaraDB RDS, you must add a rule to ApsaraDB RDS that allows access from the security group that you created in Step 2. If the service that you want to access supports only IP addresses instead of security groups, you must add the entire CIDR block of the vSwitch where the target service resides.
Configure the security group for the Hadoop cluster.
Configure the following information for the security group of the Hadoop cluster to ensure that MaxCompute can access the cluster.
Configure the inbound access rules for the security group in which the Hadoop cluster resides.
The authorization object is the security group that you created in Step 2 for the ENI.
Hive Metastore port: 9083
HDFS NameNode port: 8020
HDFS DataNode port: 50010
For example, when you connect to a Hadoop cluster that is created on Alibaba Cloud E-MapReduce, you must configure the security group rules as described above. For more information, see Create a security group.
Configure the security group for the HBase cluster.
Add the security group that is created for MaxCompute or the ENI IP address to the security group or IP address whitelist of the HBase cluster.
For example, when you connect to an Alibaba Cloud HBase cluster:
Log on to the HBase Management Console. In the upper-left corner, select a region.
In the navigation pane on the left, select Clusters.
On the Clusters page, click the name of the target cluster.
In the navigation pane on the left, select Access Control.
On the Whitelist Setting and Security Group tabs, you can click Add Whitelist or Add Security Group. If you cannot add a security group, add the ENI's IP address on the Whitelist Setting tab. Because ENI IP addresses can change if the MaxCompute configuration is modified, we recommend that you add the CIDR block of the vSwitch to the whitelist.
For more information about how to add a security group or IP address whitelist, see Set whitelists and security groups.
Configure the security group for ApsaraDB RDS.
Add the security group that is created for MaxCompute or the ENI IP address to the ApsaraDB RDS security group or IP address whitelist.
For example, when you connect to an ApsaraDB RDS instance:
Sign in to the RDS console.
In the left navigation bar, select Instances, and in the top-left corner, select the region.
In the left navigation bar, click Whitelist and SecGroup.
On the Whitelist Settings and Security Group tabs, add an IP address whitelist or a security group. Because the ENI IP address may change if the MaxCompute configuration is modified, we recommend that you add the CIDR block of the vSwitch to the whitelist.
For more information about how to add a security group or an IP address whitelist, see Configure a security group or Configure an IP address whitelist.
Step 3: Access VPC resources
After you create the direct network connection, you must add the following configurations to access the VPC network using SQL or Spark.
For other types of jobs, adjust the configurations based on the job type.
SQL access
To access a VPC network by using a UDF, run the following code. For more information, see Access VPC resources by using a UDF.
-- Set the network connection name. This is the name configured for the direct network connection. This setting is valid only for the current session. SET odps.session.networklink=testLink;To access a VPC network by using a foreign table, run the following code.
-- Set the parameter in the CREATE TABLE statement. TBLPROPERTIES( 'networklink'='<networklink_name>')To configure a network link for a data lakehouse, see Data Lakehouse 2.0 User Guide.
Spark access
After you set up the leased-line network connection, add the following configuration items to your spark-defaults.conf file or DataWorks configuration to allow Spark to access the VPC:
spark.hadoop.odps.cupid.eni.enable = true
# The format is regionid:vpcid, where vpcid is the ID of the target VPC used when creating the network connection.
spark.hadoop.odps.cupid.eni.info = regionid:vpc-**********(Optional) Step 4: Configure the whitelist
If your server has access control enabled, you must add the security group for the direct network connection to the server's whitelist.