Elastic High Performance Computing:Create a hybrid cloud cluster

How it works

Your on-premises cluster handles job scheduling as usual. E-HPC provisions cloud compute nodes and registers them with your existing scheduler, so jobs flow between on-premises and cloud nodes transparently. Network connectivity between your data center and the virtual private cloud (VPC) — provided by VPN Gateway, Cloud Enterprise Network (CEN), or Express Connect — carries all traffic between the scheduler and the cloud nodes.

Optionally, enable Proxy Mode to deploy a proxy node in the cloud. Use this option when cloud compute nodes cannot reach your on-premises scheduling node directly. The proxy node relays communication between the scheduler and the cloud side.

Prerequisites

Before you begin, ensure that you have:

• A VPC and a vSwitch. See Create and manage a VPC and Create and manage a vSwitch.

• A service-linked role for E-HPC. The first time you log on to the E-HPC console, you are prompted to create this role.

• Network connectivity between your data center and Alibaba Cloud, through one of: a VPN Gateway, a CEN instance, or an Express Connect circuit. See Create a CEN instance, Create a VPN gateway, and What is a connection over an Express Connect circuit?.

On-premises cluster requirements

The management nodes of your on-premises cluster must meet the following requirements:

• Operating system: CentOS 7.2, 7.3, 7.4, 7.5, 7.6, or 8.0

• Scheduler: Deadline, Portable Batch System (PBS) 18.1.1, Open Grid Scheduler (SGE), or a custom scheduler

  If your cluster uses PBS or SGE, the operating system must be CentOS 7.x.

• Account management: A custom account management service

Step 1: Create a hybrid cloud cluster

1. Log on to the E-HPC console.

2. In the top navigation bar, select a region.

3. Click Cluster in the top navigation bar.

4. On the Cluster page, click Create Hybrid Cluster.

5. Configure hardware settings.

   1. Set the basic parameters:

      Parameter	Description
      Availability Zone	The availability zone where cloud compute nodes are provisioned.
      Pricing Model	The billing method for cloud compute nodes. Does not apply to elastic IP addresses (EIPs) or File Storage NAS file systems. Options: Subscription (weekly, monthly, or yearly), Pay-As-You-Go (billed per hour), or Preemptible Instance (lower cost, price fluctuates with supply and demand, billed by usage duration). See ECS billing method overview.
      Compute Node	The instance type for cloud compute nodes. Overall cluster performance depends on this setting.
      Proxy Mode	When enabled, a proxy node is created in the cloud to relay communication between the on-premises scheduler and the cloud compute nodes. Enable this if cloud nodes cannot reach your scheduling node directly.
      Proxy Node	The instance type for the proxy node. Required when Proxy Mode is enabled.

   2. Expand Advanced Configurations and set network and storage parameters:

      Category	Parameter	Description
      Network configuration	VPC and vSwitch	The VPC and vSwitch where the cluster resides. The first VPC and vSwitch in each list are selected by default. Make sure the vSwitch has enough available IP addresses for all cluster nodes. You can click Create VPC and Create vSwitch (for subnet) to create a VPC and a vSwitch. See Create and manage a VPC and Create and manage a vSwitch.
      	Create Security Group and Security Group Name	A security group is created automatically and cannot be disabled. Enter a name for the new security group.
      Storage	Storage Path	How storage is mounted on the cloud compute nodes. Options: Do Not Mount File System, Use On-premises File System, or Use Cloud File System.
      	Local Cluster Storage Mount Point	The on-premises storage path (for on-premises file system).
      	Mount Path	The path on the cloud nodes where storage is mounted (for on-premises file system).
      	Local Path	The on-premises path to mount into the cluster (for on-premises file system).
      	Configure by Directory	Whether to mount different file systems for each folder (for cloud file systems).
      	Type	The cloud file system type.
      	File System ID and Mount Point	The file system and mount target to use. The first items in each list are selected by default. Click Create a file system or Create mount point to create new ones. See Create a file system and Manage mount targets.
      	Mount Configurations	For General-purpose NAS file systems, select a mount protocol: Mount over NFSv3 or Mount over NFSv4.
      	Remote Directory	The remote directory of the file system to mount.

6. Click Next.

7. Configure on-premises cluster settings:

   Parameter	Description
   Image Type and Image	The image type and image used to set up the cloud compute node environment.
   Scheduler	The scheduler type used by your on-premises cluster.
   Scheduler Node IP Address	The IP address of the scheduling node in your on-premises cluster.
   Scheduling Node Hostname	The hostname of the scheduling node in your on-premises cluster.
   Account System	The domain account service of the scheduling node in your on-premises cluster.
   Local Cluster Domain Name	The domain name of your on-premises cluster.
   Account Node IP Address	The IP address of the domain account node in your on-premises cluster.
   Account Node Hostname	The hostname of the domain account node in your on-premises cluster.

8. Click Next.

9. Configure basic cluster settings:

   Parameter	Description
   Cluster Name	The name displayed on the Cluster page.
   Logon Password	The password for SSH access to the logon node of the cluster. The username is root. If Proxy Mode is enabled, enter the root password of the on-premises cluster.
   Repeat Password	Confirm the password.

10. In the Configuration List section, review all settings. Read and accept Alibaba Cloud International Website Product Terms of Service, then click OK.

Verify: On the Cluster page, wait for the cluster status to change to Running. This indicates the cluster was created successfully.

Step 2: Scale out the hybrid cloud cluster

A newly created hybrid cloud cluster has no compute nodes. Scale it out to add cloud compute nodes.

Before scaling out, prepare a custom image with your scheduler installed. For the Deadline scheduler or a custom scheduler, the image must include the scheduler software. See Create a custom image from an instance.

1. On the Cluster page, find your cluster and click Resize.

2. In the Resize panel, complete the basic configurations. For parameter details, see Manually scale out a cluster. Set the following:

   • Image Type: Select Custom Image.

   • Image: Select the custom image you prepared.

3. Review the settings. Read and accept Alibaba Cloud International Website Product Terms of Service, then click Buy Now.

Verify: In the left navigation pane, choose Resource Management > Nodes. Select your cluster from the Cluster drop-down list and Compute Node from the Node Type drop-down list. When all nodes show Running, the scale-out is complete.