Batch Compute (Deprecated):Mount NAS

Alibaba Cloud File Storage (NAS) is a file storage service for compute nodes, such as Alibaba Cloud ECS instances, High-Performance Computing (HPC), and Docker. NAS provides standard file access protocols. This distributed file system features unlimited capacity and performance scalability, a single namespace, multi-share support, high reliability, and high availability. You can use NAS without modifying your existing applications. BatchCompute users can use NAS by configuring its related information in the API or software development kit (SDK).

1. Key concepts

For security, performance, and ease of use, BatchCompute supports NAS only within a virtual private cloud (VPC). The following are essential concepts related to VPC and NAS:

• Virtual Private Cloud (VPC): A VPC is a custom private network that you create on Alibaba Cloud. VPCs are logically isolated from each other at Layer 2. Within a VPC, you can create and manage cloud product instances, such as ECS, Server Load Balancer, RDS, and NAS.

• File system: A file system is the basic resource unit for NAS. Storage capacity, pricing, quota limits, and mount targets are all associated with the file system. For more information, see how to create a file system.

• Mount target: A mount target is an access endpoint for a file system in a VPC or classic network. Each mount target corresponds to a domain name. To mount the NAS file system, you must specify the domain name of the mount target. For more information, see how to create a mount target. Because BatchCompute supports NAS only in VPCs, you must set the mount target type to VPC when you create a mount target.

These three concepts are critical when you use NAS with BatchCompute. For more information about purchasing and using NAS or VPC, see NAS documentation and VPC documentation.

2. Use NAS

To use NAS with BatchCompute, follow these steps:

2.1. Data preparation

• Create a VPC: You can use an existing VPC or create a new one. For more information, see Create a VPC. The system automatically generates a VPC ID, for example, vpc-m5egk1jsm3qkbxxxxxxxx.

• Create a vSwitch: You can use an existing vSwitch or create a new one. For more information, see Create a vSwitch. The system automatically generates a vSwitch ID, for example, vsw-2zeue3c2rciybxxxxxxxx. You must set the vSwitch CIDR block as a subnet of the VPC CIDR block.

• Create a NAS file system: In the NAS console, create a file system as described in the documentation. The system automatically generates a file system ID, for example, 0266e49fea. BatchCompute supports only Network File System (NFS) file systems.

• Create a mount target: Set the mount target type to VPC and select the VPC and vSwitch that you created. You can use the default permission group unless you have special requirements. After the mount target is created, you can view the mount path on the mount target management page. The mount path is similar to 0266e49fea-yio75.cn-beijing.nas.aliyuncs.com.

2.2. Create a cluster in your VPC

• VpcId: The ID of the VPC where your NAS mount target resides. Replace the example ID with your actual VPC ID.

• CidrBlock: The private IP address range for the BatchCompute cluster within the VPC. This must be a subnet of your VPC CIDR block. Enter a value that aligns with your network planning. For more information, see CIDR block planning.

2.2.1. Use the Python SDK

# Specify the VPC and CIDR block when creating a cluster

cluster_desc['Configs']['Networks']['VPC']['VpcId'] = 'vpc-m5egk1jsm3qkbxxxxxxxx'
cluster_desc['Configs']['Networks']['VPC']['CidrBlock'] = '192.168.0.0/20'

# You can also specify the VPC and CIDR block when submitting an AutoJob
job_desc['DAG']['Tasks']['my-task']['AutoCluster']['Configs']['Networks']['VPC']['VpcId'] = 'vpc-m5egk1jsm3qkbxxxxxxxx'
job_desc['DAG']['Tasks']['my-task']['AutoCluster']['Configs']['Networks']['VPC']['CidrBlock'] = '192.168.0.0/20'

2.2.2. Use the Java SDK

Configs configs = new Configs();
Networks networks = new Networks();
VPC vpc = new VPC();
vpc.setCidrBlock("192.168.0.0/20");
vpc.setVpcId("vpc-m5egk1jsm3qkbxxxxxxxx");
networks.setVpc(vpc);
configs.setNetworks(networks);

// Specify NAS file system and permission group info when creating a cluster
ClusterDescription clusterDescription = new ClusterDescription();
clusterDescription.setConfigs(configs);

// You can also specify NAS file system and permission group info when submitting an AutoJob
AutoCluster autoCluster = new AutoCluster();
autoCluster.setConfigs(configs);

2.3. Specify the mount target

• Source: The source path. The path must be prefixed with nas:// and followed by the NAS mount target and the directory path of the file system. Note the following:

  • BatchCompute supports multiple mount sources. To distinguish NAS NFS file systems from other types of file systems, you must prefix the source path with nas://.

  • For Windows mounts, you must append an exclamation mark (!) to the path because of the behavior of the Windows NFS client. For more information, see Windows mount considerations.

  • You can also manually mount the file system in your application. To do this, you must place the cluster in your VPC as described in the preceding section. For more information, see Manual mount.

• Destination: The local directory path on the BatchCompute cluster nodes. You do not need to create this directory in advance. BatchCompute automatically creates it.

• WriteSupport: Specifies whether to enable write access. If you set this parameter to False, BatchCompute uses its proprietary distributed cache to improve the performance of NAS access.

2.3.1. Use the Python SDK

# Cluster-level mount
# For Linux
cluster_desc['Configs']['Mounts']['Entries'] = {
    'Source': 'nas://0266e49fea-yio75.cn-beijing.nas.aliyuncs.com:/',
    'Destination': '/home/admin/mydir/',
    'WriteSupport': true,
}

# For Windows
cluster_desc['Configs']['Mounts']['Entries'] = {
    'Source': 'nas://0266e49fea-yio75.cn-beijing.nas.aliyuncs.com:/!',
    'Destination': 'E:',
    'WriteSupport': true,
}

# Job-level mount
# For Linux
job_desc['DAG']['Tasks']['my-task']['Mounts']['Entries'] = {
    'Source': 'nas://0266e49fea-yio75.cn-beijing.nas.aliyuncs.com:/',
    'Destination': '/home/admin/mydir/',
    'WriteSupport': true,
}


# For Windows
job_desc['DAG']['Tasks']['my-task']['Mounts']['Entries'] = {
    'Source': 'nas://0266e49fea-yio75.cn-beijing.nas.aliyuncs.com:/!',
    'Destination': 'E:',
    'WriteSupport': true,
}

2.3.2. Using the Java SDK

MountEntry mountEntry = new MountEntry();
mountEntry.setSource("nas://0266e49fea-yio75.cn-beijing.nas.aliyuncs.com:/");
mountEntry.setDestination("/home/admin/mydir/");
mountEntry.setWriteSupport(true);
mounts.addEntries(mountEntry);

3. NAS mount considerations

3.1. Capacity of permission groups

A single NAS permission group supports up to 300 rules. BatchCompute supports only classic networks, where each VM uses one permission rule. Therefore, the total number of cluster instances using NAS at the same time cannot exceed 300. Exceeding this limit results in undefined behavior.

3.2. NAS billing

There are no additional fees for using NAS with BatchCompute. NAS usage is billed separately. For pricing details, see NAS pricing.

3.3. Priority of cluster and job mounts

You can specify NAS mounts in the Mounts field when you create a cluster or submit a job. BatchCompute applies the following priority rules:

• Mount entries in a job override those in the cluster (including AutoCluster), but they do not affect the file system or permission group settings for cluster mounts.

• After a job ends, cluster-level mount entries are not restored.