use OSS accelerator together with ossutil, use OSS accelerator together with ossfs, use OSS accelerator together with OSS Connector for AI and ML, use OSS accelerator together with OSS SDKs - Object Storage Service

The Object Storage Service (OSS) accelerator feature enables you to create an accelerator that works with OSS tools and OSS SDKs to efficiently perform operations such as uploading and downloading data. This topic describes how to configure and use OSS accelerator with OSS tools and OSS SDKs.

Solutions

You can use the OSS accelerator feature together with the following tools and OSS SDKs to implement data access with millisecond-level response, low latency, and high throughput:

Use OSS accelerator together with ossutil: Add the accelerated endpoint of an accelerator to an ossutil configuration file.
Use OSS accelerator together with ossfs: When you use ossfs to mount a bucket to a local directory, add the accelerated endpoint of an accelerator to the mount command.
Use OSS accelerator together with OSS Connector for AI/ML: When you use OSS Connector to build a dataset, you can use the endpoint parameter to pass the accelerated endpoint of the accelerator.
Use OSS accelerator together with OSS SDKs: OSS SDK for Java is used as an example. When you initialize the ossClient instance, the accelerated endpoint of the accelerator is passed by using the endpoint parameter.

Important

When you use OSS accelerator, set the endpoint that you use to access OSS to the accelerated endpoint of an accelerator. Examples:

When you configure ossutil, set the endpoint parameter to the accelerated endpoint of the accelerator.
When you use ossfs to mount a bucket to a local directory, set the -ourl parameter to the accelerated endpoint of the accelerator.

Prerequisites

An accelerator is created.

Solution 1: Use OSS accelerator together with ossutil

Use OSS accelerator together with ossutil

In this example, ossutil is installed on a 64-bit Linux x86 operating system. For more information about how to install ossutil on other operating systems, see Install ossutil.

Install ossutil.
1. Download the ossutil installation package.
```
curl -o ossutil-2.0.3-beta.09261200-linux-amd64.zip https://gosspublic.alicdn.com/ossutil/v2-beta/2.0.3-beta.09261200/ossutil-2.0.3-beta.09261200-linux-amd64.zip
```
2. Go to the directory in which the ossutil installation package is stored and run the following unzip command:
```
unzip ossutil-2.0.3-beta.09261200-linux-amd64.zip
```
3. Go to the directory to which the package is decompressed and run the following command:
```
chmod 755 ossutil
```
4. Run the following command in the current directory to allow ossutil to be globally called:
```
sudo mv ossutil /usr/local/bin/ && sudo ln -s /usr/local/bin/ossutil /usr/bin/ossutil
```
5. Run the following command. If the version number of ossutil is returned, ossutil 2.0 is installed.
```
ossutil version
```

Configure ossutil and add the accelerated endpoint of the accelerator.

Create an ossutil configuration file. You can specify the path and name of the configuration file.
```
touch ossutilconfig
```

Configure parameters and save the configuration file. For more information, see Configure ossutil.

#[profile dev]                  The name of the section. You can specify the name of the section.
#accessKeyId, accessKeySecret  The AccessKey ID and AccessKey secret of the RAM user. 
#region                        The region in which the bucket is located.
# buckets=dev-bucket            Use the parameters in the dev-bucket section.
#[buckets dev-bucket]          The name of the section. You can specify the name of the section.
#bucketname                    The name of the bucket. 
#endpoint                      The accelerated endpoint of the accelerator. 

[profile dev]
accessKeyId=LTA*********************ICT
accessKeySecret=V0qK**********************OOKp
region=cn-hangzhou
buckets=dev-bucket

[buckets dev-bucket]
bucket-name =
#Specify the endpoint of the accelerator.
  endpoint=cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com

Use the accelerator.
In the following example, objects are downloaded from a bucket to the test directory more quickly because an accelerator is configured for ossutil. For more information about ossutil commands, see ossutil commands.
The cp command is used to download objects. The -r parameter performs recursive operations on objects in the bucket-name bucket. The -c parameter specifies the path of the ossutil configuration file. The --profile parameter references the configurations in the dev section.
```
ossutil cp -r oss://bucket-name/ ./test/ -c ./ossutilconfig --profile dev
```

Solution 2: Use OSS accelerator together with ossfs

Use OSS accelerator together with ossfs

In this example, ossfs is installed on a Ubuntu 16.04 x86 operating system. For more information about how to install ossfs of a different version on a different operating system, see Install ossfs.

Install ossfs.

sudo wget https://gosspublic.alicdn.com/ossfs/ossfs_1.91.3_ubuntu16.04_amd64.deb
sudo apt-get update
sudo apt-get install gdebi-core
sudo gdebi ossfs_1.91.3_ubuntu16.04_amd64.deb

Add the mime.types file.
Run the following command to add the mime.types file to ensure that the content type of the uploaded object matches the object name extension:
```
sudo apt-get install mime-support
```
Configure ossfs, use ossfs to mount a bucket to the local file system, and configure an accelerator for the bucket. For more information about how to perform mount operations by using ossfs, see Configure and mount ossfs.
1. Store the AccessKey ID and AccessKey Secret in the account configuration file named /etc/passwd-ossfs.
```
echo bucket-test:LTAI****************2CT:V0************************OKp > /etc/passwd-ossfs
```
2. Set the permissions of the configuration file to 640.
```
chmod 640 /etc/passwd-ossfs 
```
3. Create a mount directory.
```
mkdir bucket-test
```
4. Mount a bucket named bucket-test to the /tmp/bucket-test/ directory and specify the accelerated endpoint of the accelerator.
```
ossfs bucket-test /tmp/bucket-test/ -ourl=cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com
```
Use the accelerator.
After an accelerator is configured for ossfs, you can efficiently perform read and write operations on objects in the bucket that is mounted to the local file system. For example, you can load model files stored in OSS more quickly in a local model training task.

Solution 3: Use OSS accelerator together with OSS Connector for AI or ML

Use OSS accelerator together with OSS Connector for AI or ML

The following example shows how to install OSS Connector for AI/ML for Python 3.12. To install OSS Connector for AI/ML for Python of a different version, change the Python version to a specific version number.

Prerequisites:

Operating system: 64-bit x86 Linux
glibc: 2.17 or later
Python: 3.8 to 3.12
PyTorch: 2.0 or later
To use the OSS checkpoint feature, the Linux kernel must support userfaultfd.
Note
In this example, Ubuntu is used. You can run the sudo grep CONFIG_USERFAULTFD /boot/config-$(uname -r) command to check whether the Linux kernel supports userfaultfd. If CONFIG_USERFAULTFD=y is returned, the Linux kernel supports userfaultfd. If CONFIG_USERFAULTFD=n is returned, the Linux kernel does not support userfaultfd. This means that the OSS Checkpoint feature cannot be used.

Install OSS Connector for AI/ML.
1. Run the pip3.12 install osstorchconnector command to install OSS Connector for AI/ML in the container that is generated by using Linux or an image based on Linux.
```
pip3.12 install osstorchconnector
```
2. Run the pip3.12 show osstorchconnector command to check whether the OSS Connector for AI/ML is installed.
```
pip3.12 show osstorchconnector
```
  If the version information of osstorchconnector is returned, OSS Connector for AI/ML is installed.
Configure OSS Connector for AI/ML.
1. Create a configuration file for access credentials.
```
mkdir -p /root/.alibabacloud && touch /root/.alibabacloud/credentials
```
2. Add the access credentials to the configuration file and save the configuration file.
  Replace <Access-key-id> and <Access-key-secret> in the example with the AccessKey ID and AccessKey secret of a RAM user. For more information about how to create an AccessKey ID and AccessKey secret, see Create an AccessKey pair. For more information about the configuration items and configuration by using temporary access credentials, see Configure access credentials.
```
{
  "AccessKeyId": "LTAI************************",
  "AccessKeySecret": "At32************************"
}
```
3. Create an configuration file for OSS Connector.
```
mkdir -p /etc/oss-connector/ && touch /etc/oss-connector/config.json
```
4. Add the configurations of the OSS connector to the configuration file and save the configuration file. For more information about the configuration items, see Configure OSS Connector.
  In most cases, you can use the following default configurations.
```
{
    "logLevel": 1,
    "logPath": "/var/log/oss-connector/connector.log",
    "auditPath": "/var/log/oss-connector/audit.log",
    "datasetConfig": {
        "prefetchConcurrency": 24,
        "prefetchWorker": 2
    },
    "checkpointConfig": {
        "prefetchConcurrency": 24,
        "prefetchWorker": 4,
        "uploadConcurrency": 64
    }
}
```

Use the accelerator.

The following sample code provides an example on how to use the from_prefix method of OssMapDataset to build a Dataset more quickly from the cache capacity of the accelerator. ENDPOINT specifies the accelerated endpoint of the accelerator.

from osstorchconnector import OssMapDataset

# Replace ENDPOINT with the accelerated endpoint of the accelerator. 
ENDPOINT = "cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com"
# Specify the path of the OSS Connector for AI/ML configuration file. 
CONFIG_PATH = "/etc/oss-connector/config.json"
# Specify the path of the access credential configuration file. 
CRED_PATH = "/root/.alibabacloud/credentials"
# Specify the path of the directory of the bucket. 
OSS_URI = "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/"

# Use the from_prefix method of OssMapDataset to build a dataset.
map_dataset = OssMapDataset.from_prefix(oss_uri=OSS_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)

# Access objects in the created dataset at random.
item = map_dataset[0]
print(item.key)
content = item.read()
print(item.size)
print(len(content))

# Traverse the objects in the dataset.
for item in map_dataset:
    print(item.key)
    print(item.size)
    content = item.read()
    print(len(content))

Solution 4: Use OSS accelerator together with OSS SDKs

Use OSS accelerator together with OSS SDKs

In the following example, OSS accelerator works with OSS SDK for Java. If you want to use OSS accelerator together with OSS SDK for other programming languages, specify the accelerated endpoint of the accelerator when you initialize the OSSClient instance.

Prerequisites:
- Java 1.7.0 or later is installed.
- You can run the java -version command to check the Java version.

The dependencies for OSS SDK for Java are added to the Maven project. For more information about how to import the JAR package to OSS SDK for Java, see Install SDK.

To use OSS SDK for Java in Maven, you need to only add the required dependencies to the pom.xml file. In this example, OSS SDK for Java 3.17.4 is added.

<dependency>
    <groupId>com.aliyun.oss</groupId>
    <artifactId>aliyun-sdk-oss</artifactId>
    <version>3.17.4</version>
</dependency>

If you use Java 9 or later, add JAXB-related dependencies. The following code provides an example on how to add jaxb-related dependencies:

<dependency>
    <groupId>javax.xml.bind</groupId>
    <artifactId>jaxb-api</artifactId>
    <version>2.3.1</version>
</dependency>
<dependency>
    <groupId>javax.activation</groupId>
    <artifactId>activation</artifactId>
    <version>1.1.1</version>
</dependency>
<!-- no more than 2.3.3-->
<dependency>
    <groupId>org.glassfish.jaxb</groupId>
    <artifactId>jaxb-runtime</artifactId>
    <version>2.3.3</version>
</dependency>

Use the accelerator.

The following sample code provides an example on how to use OSS SDK for Java together with an accelerator to upload an object. Replace the endpoint value with the accelerated endpoint of the accelerator.

import com.aliyun.oss.ClientBuilderConfiguration;
import com.aliyun.oss.OSS;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.common.auth.CredentialsProviderFactory;
import com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider;
import com.aliyun.oss.common.comm.SignVersion;

import java.io.ByteArrayInputStream;

public class OssCliectTest {
    public static void main(String[] args) throws Exception {
        // Specify the accelerated endpoint of the accelerator.
        String endpoint = "cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com";
        // Specify the region in which the bucket is located. Example: cn-hangzhou.
        String region = "cn-hangzhou";

        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the environment variables are configured.
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();

        // Specify the name of the bucket. Example: examplebucket.
        String bucketName = "BucketName";
        // Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full paths.
        String objectName = "Test.txt";

        // Create an OSSClient instance.
        // Call the shutdown method to release resources when the OSSClient is no longer in use.
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        // Explicitly declare the use of the V4 signature algorithm
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
                .endpoint(endpoint)
                .credentialsProvider(credentialsProvider)
                .clientConfiguration(clientBuilderConfiguration)
                .region(region)
                .build();

        String content = "##################";
        // Upload the configurations of the content variable to the Test.txt file in the bucket.
        ossClient.putObject(bucketName,objectName,new ByteArrayInputStream(content.getBytes()));

        // Shut down the OSSClient instance.
        ossClient.shutdown();
    }
    
  }