When you download multiple objects from Object Storage Service (OSS) at a time, inconvenience may be caused if the sizes of the objects are small and the number of the objects is great. You can use Function Compute to package and download multiple objects at a time from OSS to your on-premises device. This saves time, traffic, and costs. This topic describes how to use Function Compute to download multiple objects from OSS at a time.

Prerequisites

Before you begin, make sure that you have completed the following operations:

Process

Figure 1. Flowchart
solution
  1. Invoke a function as needed and specify an OSS bucket and the objects to be compressed.
  2. After you invoke the function, Function Compute downloads the objects from OSS and compress the objects into a ZIP package.
  3. Function Compute uploads the ZIP package to OSS.
  4. Function Compute returns the URL of the ZIP package in OSS.
  5. Download the ZIP package by using the URL that is returned in Step 4.
Figure 2. Implementation details

The following figure shows the details of Step 2 and Step 3 in the flowchart.

principle
This process has the following benefits:
  • Function Compute relieves the pressure of running functions in limited disk space. Streaming download and upload are used to transfer the ZIP package. This way, only a small amount of data is cached in the buffer.
  • Function Compute generates the ZIP package and upload the generated ZIP package to OSS at the same time. This reduces the time that is taken to upload and download objects.
  • When Function Compute uploads the ZIP package to OSS, the multipart upload feature of OSS is used to concurrently upload parts in queues in multi-threaded mode.

Procedure

  1. Use one of the following methods to download the sample project:
    Notice If you are using the Windows operating system, download cURL and double-click the installer to install cURL.
    • If you have installed Git, run the following command on the terminal:
      git clone https://github.com/awesome-fc/zip-oss.git
    • If you have not installed Git, download the code package.
  2. In the zip-oss project, run the fun deploy command to deploy the project and obtain the URL.
    fcimageoss
  3. After the project is deployed, invoke a deployed function to package objects.
    1. In the zip-oss folder, modify the values of the parameters in the event.json file.
      cat <<EOF > event.json
      {
        "region": "{region}", // Replace {region} with the region where the OSS bucket resides, such as cn-shanghai.
        "bucket": "{bucket name}", // Replace {bucket name} with the name of the OSS bucket.
        "source-dir": "{filepath/}" // Replace {filepath/} with the directory of the objects to be compressed.
      }
      EOF
    2. Run the curl command to invoke the relevant function.
      In the zip-oss folder, create a folder to store the decompressed objects. You can customize the name of the folder, such as backfile. Then, run one of the following commands based on your operating system:
      • macOS and Linux
        Command syntax:
        curl -v -L -o /localpath/oss.zip -d @./event.json urlvalue
        Command used in this example:
        curl -v -L -o /backfile/oss.zip -d @./event.json https://18807708****3420.cn-beijing.fc.aliyuncs.com/2016-08-15/proxy/zip-service/zip-oss/
      • Windows
        Command syntax:
        curl -v -o ./localpath/oss.zip -d /event.json urlvalue
        Command used in this example:
        curl -v -o ./backfile/oss.zip -d /event.json https://18807708****3420.cn-beijing.fc.aliyuncs.com/2016-08-15/proxy/zip-service/zip-oss/
      After the function is invoked, open the zip-oss\backfile directory. All the objects in the backfile directory are objects stored in the specified OSS bucket.

Sample data

Scenario Number of objects Total size before compression Total size after compression Execution time
1 7 1.2 MB 1.16 MB 0.4s
2 57 1.06 GB 1.06 GB 63s
You can obtain the following information from the preceding table:
  • Scenario 1 shows that when you use Function Compute to download multiple objects, the storage space occupied by the objects is reduced.
  • Scenario 2 shows that when you use Function Compute to download multiple objects, a large number of objects can be downloadedin a short period of time.
Note The preceding table is only for reference.