Read write OSS data PAI DSW Python SDK API - Platform For AI

Use the OSS Python SDK or the OSS Python API to read and write Object Storage Service (OSS) data in DSW.

Recommendations

For frequent large-scale data access, register OSS as a dataset and mount it. For occasional or logic-dependent access, use the SDK and API methods in this topic.

Use the OSS Python SDK

DSW includes the oss2 Python package. To read and write OSS data:

Authenticate and initialize the client.

import oss2
auth = oss2.Auth('<your_AccessKey_ID>', '<your_AccessKey_Secret>')
bucket = oss2.Bucket(auth, '<your_oss_endpoint>', '<your_bucket_name>')

Replace these placeholders.

Parameter	Description
<your_AccessKey_ID> and <your_AccessKey_Secret>	The AccessKey ID and AccessKey secret for your Alibaba Cloud account. For more information, see Create an AccessKey.
<your_oss_endpoint>	The OSS endpoint. Select the endpoint for your instance region: Pay-as-you-go instances in the China (Beijing) Region: `oss-cn-beijing.aliyuncs.com` Subscription instances in the China (Beijing) region: `oss-cn-beijing-internal.aliyuncs.com` GPU P100 instances or CPU instances in the China (Shanghai) region: `oss-cn-shanghai.aliyuncs.com` GPU M40 instances in the China (Shanghai) region: `oss-cn-shanghai-internal.aliyuncs.com` Regions and endpoints of OSS.
<your_bucket_name>	The bucket name, without the `oss://` prefix.

Read and write OSS data.

# Read a complete file.
result = bucket.get_object('<your_file_path/your_file>')
print(result.read())
# Read data by range.
result = bucket.get_object('<your_file_path/your_file>', byte_range=(0, 99))
# Write data to OSS.
bucket.put_object('<your_file_path/your_file>', '<your_object_content>')
# Append data to a file.
result = bucket.append_object('<your_file_path/your_file>', 0, '<your_object_content>')
result = bucket.append_object('<your_file_path/your_file>', result.next_position, '<your_object_content>')

Replace these placeholders:

<your_file_path/your_file>: The path to the file you want to read or write.
<your_object_content>: The content you want to write or append.

Use the OSS Python API

DSW provides the OSS Python API for PyTorch users to read and write OSS data directly.

Store training data or models in OSS:

Load training data

Store data in an OSS bucket with an index file that maps paths to labels. Create a custom Dataset to use the PyTorch DataLoader API for multi-process parallel reads. Example:

import io
import oss2
import PIL
import torch
class OSSDataset(torch.utils.data.dataset.Dataset):
    def __init__(self, endpoint, bucket, auth, index_file):
        self._bucket = oss2.Bucket(auth, endpoint, bucket)
        self._indices = self._bucket.get_object(index_file).read().split(',')
    def __len__(self):
        return len(self._indices)
    def __getitem__(self, index):
        img_path, label = self._indices(index).strip().split(':')
        img_str = self._bucket.get_object(img_path)
        img_buf = io.BytesIO()
        img_buf.write(img_str.read())
        img_buf.seek(0)
        img = Image.open(img_buf).convert('RGB')
        img_buf.close()
        return img, label
dataset = OSSDataset(endpoint, bucket, auth, index_file)
data_loader = torch.utils.data.DataLoader(
    dataset,
    batch_size=batch_size,
    num_workers=num_loaders,
    pin_memory=True)

Replace these placeholders:

endpoint: The OSS endpoint.
bucket: The Bucket name.
auth: The Authentication object.
index_file: The path to the index file.

Note

Index file format: commas (,) separate samples, colons (:) separate the path from the label.

Save or load a model
Save or load a PyTorch model with the oss2 Python API. PyTorch serialization tutorial.
- Save a model
```
from io import BytesIO
import torch
import oss2
# Specify the Bucket name.
bucket_name = "<your_bucket_name>"
bucket = oss2.Bucket(auth, endpoint, bucket_name)
buffer = BytesIO()
torch.save(model.state_dict(), buffer)
bucket.put_object("<your_model_path>", buffer.getvalue())
```
  Replace these placeholders:
  - auth: The Authentication object.
  - endpoint: The OSS endpoint.
  - <your_bucket_name>: The OSS Bucket name, without the oss:// prefix.
  - <your_model_path>: The model destination path in the bucket.
- Load a model
```
from io import BytesIO
import torch
import oss2
bucket_name = "<your_bucket_name>"
bucket = oss2.Bucket(auth, endpoint, bucket_name)
buffer = BytesIO(bucket.get_object("<your_model_path>").read())
model.load_state_dict(torch.load(buffer))
```
  Replace these placeholders:
  - auth: The Authentication object.
  - endpoint: The OSS endpoint.
  - <your_bucket_name>: The OSS Bucket name, without the oss:// prefix.
  - <your_model_path>: The model path in the bucket.