When you download objects from Object Storage Service (OSS), it may be inconvenient to download them one by one if the objects are small and you need to download a large number of objects. You can use Function Compute to compress the objects that you need to download into a package and download the package from OSS to your on-premises device. This way, you can download multiple objects at a time. This saves time, traffic, and costs. This topic describes how to use Function Compute to download multiple objects from OSS at a time.

Prerequisites

The following operations are complete:

Process

Figure 1. Flowchart
solution
  1. Invoke a function as needed and specify an OSS bucket and the objects to be compressed.
  2. After you invoke the function, Function Compute downloads the objects from OSS and compress the objects into a ZIP package.
  3. Function Compute uploads the ZIP package to OSS.
  4. Function Compute returns the URL of the ZIP package in OSS.
  5. Download the ZIP package by using the URL that is returned in Step 4.
Figure 2. Implementation details

The following figure shows the details of Step 2 and Step 3 in the flowchart.

principle
This process has the following benefits:
  • Streaming download and upload are used to transfer the ZIP package. This way, only a small amount of data is cached in the memory. This resolves the bottleneck that may be caused by the limited disk space of the function runtime environment.
  • Function Compute generates the ZIP package and uploads the generated ZIP package to OSS at the same time. This reduces the time that is taken to upload and download objects.
  • The multipart upload feature of OSS is used to concurrently upload parts of the ZIP package in queues in multi-threaded mode.

Procedure

  1. Use one of the following methods to download the sample project:
    • Method 1: Run the following command to download the ZIP package:
      git clone https://github.com/awesome-fc/zip-oss.git
    • Method 2: Visit the following URL in a browser and download the ZIP package:
      https://codeload.github.com/awesome-fc/zip-oss/zip/master
  2. Run the following command to go to the project directory:
    cd zip-oss
  3. Run the following command to deploy the project and obtain the function URL:
    s deploy -y
    Sample command output:
     Checking Service, Function, Triggers (2.44s)
     Creating Service, Function, Triggers (1.18s)
    
    ...
    fc-zip-service-zip-oss:
      region:   cn-shanghai  service:
        name: zip-service
      function:
        name:       zip-oss
        runtime:    python2.7
        handler:    main.main_handler
        memorySize: 3072
        timeout:    900
      url:
        system_url: https://188077086902****.cn-shanghai.fc.aliyuncs.com/2016-08-15/proxy/zip-service/zip-oss/
      triggers:
        -
          type: http
          name: http-test
  4. After the project is deployed, invoke the deployed function to compress objects.
    1. In the event.json file, modify the parameters.
      cat <<EOF > event.json
      {
        "region": "{region}", // Replace {region} with the ID of the region in which the OSS bucket resides, such as cn-shanghai. 
        "bucket": "{bucket name}", // Replace {bucket name} with the name of the OSS bucket. 
        "source-dir": "{filepath/}" // Replace {filepath/} with the directory of the objects to be decompressed. 
      }
      EOF
    2. Run the curl command to invoke the function.
      In the zip-oss folder, create a folder to store the decompressed objects. You can use a custom folder name, such as backfile. Then, run one of the following commands based on your operating system:
      • macOS or Linux
        Syntax:
        curl -v -L -o /localpath/oss.zip -d @./event.json urlvalue
        Command used in this example:
        curl -v -L -o /backfile/oss.zip -d @./event.json https://188077086902****.cn-beijing.fc.aliyuncs.com/2016-08-15/proxy/zip-service/zip-oss/
      • Windows
        Syntax:
        curl -v -o ./localpath/oss.zip -d /event.json urlvalue
        Command used in this example:
        curl -v -o ./backfile/oss.zip -d /event.json https://188077086902****.cn-beijing.fc.aliyuncs.com/2016-08-15/proxy/zip-service/zip-oss/
      After the function is invoked, go to the zip-oss\backfile directory. All the objects stored in the specified OSS bucket are downloaded to the backfile directory.

Sample data

Scenario Number of objects Total size before compression Total size after compression Executed time
1 7 1.2 MB 1.16 MB 0.4s
2 57 1.06 GB 1.06 GB 63s
You can obtain the following information from the preceding table:
  • Scenario 1 shows that the required storage space is reduced when you use Function Compute to download multiple objects at a time.
  • Scenario 2 shows that a large number of objects can be downloaded in a short period of time when you use Function Compute to download multiple objects at a time.
Note The preceding table is only for reference.