MaxCompute:VPC access (Direct connect)

Prerequisites

Before you begin, ensure that you have:

• A MaxCompute project. For data lakehouse use cases, set the data type edition to the Hive-compatible data type edition.

• A single Alibaba Cloud account that owns the VPC, has access to the MaxCompute project, and administers the target service. All three roles must belong to the same account.

Who needs to be involved:

This procedure spans multiple systems and may require coordination across teams:

• MaxCompute project administrator: grants permissions and creates the network connection in the MaxCompute console.

• VPC/ECS console administrator: creates and configures the security group.

• Target service administrator (ApsaraDB RDS, HBase, or Hadoop): configures access rules on the target service.

Limitations

• The security group you create for this connection must be a Basic Security Group. Advanced Security Groups deny outbound traffic by default, which prevents MaxCompute from reaching services in your VPC.

• After MaxCompute creates its Elastic Network Interface (ENI) IP addresses, those addresses may change if the MaxCompute configuration is modified. To avoid connectivity disruptions, add the CIDR block of the vSwitch rather than individual ENI IP addresses to any service whitelists.

• Do not modify or repurpose the two auto-created security groups (MaxCompute-vpc-xxx and MaxCompute-backup-vpc-xxx).

Step 1: Grant permissions

Grant the operating user permission to create network connection objects.

The user performing the authorization must be a Project Owner, or hold the tenant-level Super_Administrator or Admin role. For role details, see Role planning. For the authorization procedure, see List of object permissions in a tenant.

Authorize MaxCompute to create ENIs in your VPC.

This authorization lets MaxCompute establish network connectivity by creating Elastic Network Interfaces (ENIs) in your VPC. While logged in to your Alibaba Cloud account, click Authorize.

Step 2: Create a security group for MaxCompute

Create a dedicated Basic Security Group in your VPC. MaxCompute places its ENIs in this group to access services inside the VPC. During connection creation, MaxCompute automatically creates two ENIs based on bandwidth requirements. These ENIs are free of charge.

• Set the outbound rules for this security group to control the destination addresses that MaxCompute jobs, which run on ENIs, can access. If you have no special requirements, you can keep the default outbound rules.

• All inbound traffic to the ENI is return traffic, so allow all inbound traffic.

1. Log on to the Virtual Private Cloud (VPC) console.

2. In the left navigation pane, choose VPC. In the upper-left corner, select a region.

3. On the VPC page, click the Instance ID/Name of the target VPC.

4. On the VPC details page, click the Resource Management tab.

5. In the VPC Resources section, hover over the Security Group value and click Add. For more information, see Create a security group.

   • Set Security Group Type to Basic Security Group. > Note: A Basic Security Group allows outbound traffic by default. An Advanced Security Group denies outbound traffic by default and cannot be used here.

   • For Network, select the same VPC used by the connectivity service.

6. Configure security group rules to allow MaxCompute access. For more information, see Security group application guide and examples.

   1. In the Operation column for the new security group, click Manage Rules.

   2. On the Inbound tab in the Access Rules area, click Edit for the target rule and set the following:

      Field	Value
      Action	Allow
      Priority	1
      Protocol	All Traffic
      Source	CIDR block of the VPC or vSwitch containing the target service
      Destination (Current Instance)	ALL(-1/-1)

7. (HBase only) If HBase does not support adding security groups and only supports IP-based access: log on to the ECS console, and click Elastic Compute Service in the left navigation pane to get the ENI IP address. Because ENI IP addresses can change, add the CIDR block of the vSwitch instead.

Step 3: Create a network connection

An Alibaba Cloud account, or a RAM user with the tenant-level Super_Administrator or Admin role, can create a network connection. For more information, see MaxCompute tenant-level roles.

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the left navigation pane, choose Manage Configurations > Network Connection.

3. On the Network Connection page, click Add Network Connection.

4. In the Add Network Connection dialog box, configure the parameters and click OK.

   Important

   The first time you add a network connection, you must first grant authorization to allow MaxCompute to request network interface cards. Without this authorization, the connection cannot be created.

   Parameter	Required	Description
   Connection Name	Required	A custom name for the connection.
   Type	Required	Defaults to Passthrough.
   Region	Required	Auto-populated based on the region selected in the upper-left corner.
   VPC Selected	Required	The VPC that contains the target service. To create a new VPC, see Create or delete a VPC.
   Switch	Required	The vSwitch (subnet) within the VPC. Deploy resources across vSwitches in different zones to protect against single-zone failures. To create a vSwitch, see Create or delete a vSwitch.
   Security group	Required	The Basic Security Group you created in Step 2. This group controls which resources MaxCompute's ENIs can reach and which ports can be accessed.

Step 4: Configure access rules on the target service

After the network connection is established, add inbound rules to the target service's security group to allow traffic from the MaxCompute security group (the one created in Step 2). If the target service only supports IP-based access rather than security groups, add the CIDR block of the vSwitch.

Hadoop cluster

Add inbound rules to the Hadoop cluster's security group with the security group created in Step 2 as the authorization object. Open the following ports:

Component	Port
Hive Metastore	9083
HDFS NameNode	8020
HDFS DataNode	50010

For example, when connecting to a Hadoop cluster on Alibaba Cloud E-MapReduce, the required security group rules look like this:

For more information, see Create a security group.

Step 5: Access VPC resources from your jobs

-- Applies to the current session only
SET odps.session.networklink=testLink;

TBLPROPERTIES('networklink'='<networklink_name>')

spark.hadoop.odps.cupid.eni.enable = true
spark.hadoop.odps.cupid.eni.info=regionid:vpc id

(Optional) Step 6: Add a server whitelist

If access control is enabled on your server, add the security group created in Step 2 to the server's whitelist.