By default, MaxCompute Spark cannot access services in a virtual private cloud (VPC). To access services in a VPC, such as specific IP addresses, domain names, ApsaraDB RDS, HBase clusters, or Hadoop clusters, you must establish network connectivity between MaxCompute and the VPC.
Limitations
Account limits
The MaxCompute project that runs your Spark job and the target VPC must belong to the same Alibaba Cloud account. In other words, the UID of the Alibaba Cloud account that owns the project must match the UID of the account that owns the VPC. Otherwise, the job fails with the following error:
"You are not allowed to use this vpc - vpc owner and project owner must be the same person".This procedure describes how to connect to a single VPC. If your job needs to access multiple VPCs simultaneously, you can link the connected VPC to additional VPCs using a leased line and an Elastic Network Interface (ENI). For more information, contact Alibaba Cloud VPC technical support.
Procedure
For more information, see VPC access solution (leased-line direct connection).
Step 1: Prepare your account and project
Before you establish a network connection between MaxCompute and the target service, ensure that the following conditions are met.
Create a MaxCompute Project. If you use the data lakehouse solution, we recommend that you set the data type edition for your MaxCompute project to the Hive-compatible data type edition.
To access a target service in a VPC, ensure that the VPC owner account, the Alibaba Cloud account that is used to access the MaxCompute project, and the administrator account of the target service environment or cluster all belong to the same Alibaba Cloud account.
Step 2: Grant permissions
Grant the operating user the permission to create network connection objects.
The authorized user must be a Project Owner or a user with the tenant-level Super_Administrator or Admin role. For more information, see Role planning.
For the authorization procedure, see List of Object Permissions in a Tenant.
Authorize MaxCompute: This allows MaxCompute to create ENIs in your VPC to enable network connectivity from MaxCompute to your VPC. Click Authorize while you are logged on to your Alibaba Cloud account.
Step 3: Add security group rules
In the VPC-connected instance, you must create a separate security group. This group is used to control access from MaxCompute to resources in the VPC.
You must create a new basic security group. Do not use other types of security groups or security groups that are already in use. MaxCompute creates ENIs in your VPC to access your services and automatically places them in this security group.
Set the outbound rules for this security group to control the destination addresses that MaxCompute jobs, which run on ENIs, can access. If you have no special requirements, you can keep the default outbound rules.
Traffic that enters the ENI is return traffic. Therefore, you must allow all inbound traffic.
Log on to the Virtual Private Cloud (VPC) console.
In the navigation pane on the left, choose VPC. In the upper-left corner, select a region.
On the VPC page, click the Instance ID/Name of the target VPC.
On the VPC details page, click the Resource Management tab.
On the Resource Management tab, in the VPC Resources section, hover over the value for Security Group and click Add.
Set Security Group Type to Basic Security Group.
A Basic Security Group allows outbound traffic by default. An Advanced Security Group denies outbound traffic by default, which prevents access to services in the VPC.
Select the same VPC Network used by the connectivity service.
For more information, see Create a security group.
Configure the security group rules to allow access from MaxCompute.
In the Operation column for the target security group, click Manage Rules.
On the Inbound tab in the Access Rules area, click Edit in the Actions column for the target rule. Configure the settings to allow all Inbound traffic.
Set Action to Allow.
The Priority is set to 1.
For Protocol, select All Traffic.
Source is the CIDR block of the VPC or VSwitch that contains the Alibaba Cloud service to access.
Destination (Current Instance) defaults to ALL(-1/-1).
For more information, see Security Group Application Guide and Examples.
In an HBase scenario, if HBase cannot grant network access to a security group, you can add the Elastic Network Interface (ENI) IP address created by MaxCompute to the whitelist. Because ENI IP addresses can change, we recommend that you add the CIDR block of the vSwitch for the VPC-connected instance to the whitelist. Log on to the ECS console. In the navigation pane on the left, click Elastic Compute Service to obtain the ENI IP address.
During the network connection creation process, MaxCompute automatically creates two ENIs based on bandwidth requirements. These ENIs are free of charge and are placed in this security group.
Step 4: Create a network connection between MaxCompute and the target VPC
In the MaxCompute console, an Alibaba Cloud account or a RAM user with the Super_Administrator or Admin role at the MaxCompute tenant level can create a connection to a VPC network. For more information, see MaxCompute tenant-level roles. To create the connection, perform the following steps:
Log on to the MaxCompute console, and select a region in the upper-left corner.
In the navigation pane on the left, choose .
On the Network Connection page, click Add Network Connection.
In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.
In the Add Network Connection dialog box, configure the parameters as prompted and click OK. When you add a network connection for the first time, you must first grant authorization to allow the MaxCompute platform proxy to request network interface cards. Otherwise, the connection cannot be created.
The following table describes the parameters.
Parameter
Required
Description
Connection Name:
Required
Enter a custom name for the connection.
Type:
Required
The default value is Passthrough.
Region:
Required
The system automatically populates this parameter based on the region you selected in the upper-left corner.
VPC Selected:
Required
A virtual private cloud (VPC) is an isolated virtual network. It provides a secure and configurable private network space similar to a traditional data center.
To create a new VPC, see Create or delete a VPC.
Switch:
Required
A vSwitch defines a subnet. Service interconnection is enabled between different vSwitches in the same VPC. Deploy resources across vSwitches in different zones to protect your application from failures in a single zone.
If no vSwitch is available, see Create or delete a vSwitch.
Security group:
Required
A security group acts as a virtual firewall for your resources. Manage security groups and their rules to implement fine-grained network isolation and access control.
To create a security group, see Create a security group.
Step 5: Configure your job
After you set up the leased-line network connection, add the following configuration items to your spark-defaults.conf file or DataWorks configuration to allow Spark to access the VPC:
spark.hadoop.odps.cupid.eni.enable = true
# Format: region:vpcid. Replace vpcid with the target VPC ID used during leased-line setup.
spark.hadoop.odps.cupid.eni.info = regionid:vpc-**********
# Example: cn-hangzhou:vpc-bp1wth********04ug3s