Elastic High Performance Computing:Import job files from an OSS bucket to a cluster - Elastic High Performance Computing - Alibaba Cloud - Elastic High Performance Computing

Store your job scripts and data files in an Object Storage Service (OSS) bucket so that cluster nodes can pull them at job submission time. This topic describes how to upload job files to an OSS bucket and download them onto a cluster node.

Prerequisites

Before you begin, make sure that you have:

An E-HPC cluster. See Log on to a cluster
OSS activated in the same region as your cluster. See Activate OSS
wget or curl installed on the cluster node

Upload job files to an OSS bucket

Activate OSS in the same region as your E-HPC cluster.
Create a bucket to store your job files.
Upload your job files to the bucket. In the examples below, the script file is test.py and the job data file is test.data.

Import job files from OSS to a cluster

Log on to your cluster, then use one of the following methods to download the job files to the /ehpcdata directory.

Method 1: Download using the OSS endpoint URL

Use this method when the cluster node has network access to the OSS bucket — either through the VPC internal endpoint or over the internet.

Get the endpoint URL for your bucket:
- Internal endpoint: See Use an internal endpoint to access OSS
- Public endpoint: See Use a public endpoint to access OSS

Download the job files to /ehpcdata. If wget is available:

cd /ehpcdata
wget http://<bucket>.<internal-endpoint>/test.py
wget http://<bucket>.<internal-endpoint>/test.data

If wget is not available, use curl:

cd /ehpcdata
curl -O http://<bucket>.<internal-endpoint>/test.py
curl -O http://<bucket>.<internal-endpoint>/test.data

Method 2: Download using a signed URL

Use this method when the bucket is private and you want to grant time-limited access to specific objects without configuring an AccessKey on the cluster node.

Note

Signed URLs expire after the validity period you set. Set the shortest validity period that covers your download window to limit exposure.

Generate a signed URL for each job file. See Authorize third-party users to download objects for instructions. A signed URL follows this pattern:
```
http://<bucket>.<region>.aliyuncs.com/<object>?OSSAccessKeyId=<id>&Expires=<unix-time>&Signature=<signature>
```

Download the job files using the signed URLs. If wget is available:

cd /ehpcdata
wget "http://<bucket>.<region>.aliyuncs.com/<object>?OSSAccessKeyId=<user access_key_id>&Expires=<unix time>&Signature=<signature_string>/test.py"
wget "http://<bucket>.<region>.aliyuncs.com/<object>?OSSAccessKeyId=<user access_key_id>&Expires=<unix time>&Signature=<signature_string>/test.data"

If wget is not available, use curl:

cd /ehpcdata
curl -O "http://<bucket>.<region>.aliyuncs.com/<object>?OSSAccessKeyId=<user access_key_id>&Expires=<unix time>&Signature=<signature_string>/test.py"
curl -O "http://<bucket>.<region>.aliyuncs.com/<object>?OSSAccessKeyId=<user access_key_id>&Expires=<unix time>&Signature=<signature_string>/test.data"

Replace each placeholder with the values from the signed URL generated in step 1.