Object Storage Service:How do I query the storage size of objects with a specific object key prefix?

Methods

Use inventory

The following example procedure shows how to query the storage size of objects with the "bill/" key prefix by exporting an inventory list of the objects.

1. Configure an inventory.

   1. On the Buckets page of the OSS console, click the desired bucket.

   2. In the left-side navigation tree, choose Data Management > Bucket Inventory.

   3. On the Bucket Inventory page, click Create Inventory.

   4. In the Create Inventory panel, configure the parameters as described in the following table, retain the default settings for other parameters, and click OK.

      

      Parameter	Description
      Rule Name	Specify a name for the inventory. In this example, the name "inventory001" is used.
      Inventory Storage Bucket	Select a bucket in the current region within the same Alibaba Cloud account.
      Frequency	Select Daily.
      Optional Fields	Select Object Size.
      Object Prefix	Specify bill/.

2. Calculate the storage size of objects with the 'bill/' prefix.

   1. On the Objects page, go to the {bucketName}/{inventoryName}/data/ path and find the inventory list whose name ends with csv.gz.

      

   2. Click > Download to download the inventory list to a local path and decompress the file.

   3. Calculate the storage size.

      Open the CSV inventory list and sum the numbers in the C column. The sum is the total storage size of the objects in bytes.

3. (Optional) Delete the inventory.

   • You can have much convenience by exporting inventory lists on a daily basis. However, this practice can also increase your storage costs because inventory lists incur storage charges. We recommend that you avoid frequently exporting inventory lists to prevent unnecessary costs. In scenarios where you do not require frequent generation of inventory lists, you can delete the inventory after you obtain the first inventory list.

     

List objects

import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.CredentialsProviderFactory;
import com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider;
import com.aliyun.oss.common.comm.SignVersion;
import com.aliyun.oss.model.*;

import java.util.List;

public class Demo {
    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Specify the ID of the region that maps to the endpoint. Example: cn-hangzhou.
        String region = "cn-hangzhou";
        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();

        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "examplebucket";
        // Specify the prefix. 
        String keyPrefix = "bill/";
        
        // Create an OSSClient instance. 
        // Call the shutdown method to release resources when the OSSClient is no longer in use.
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        // Explicitly declare the use of the V4 signature algorithm.
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
                .endpoint(endpoint)
                .credentialsProvider(credentialsProvider)
                .clientConfiguration(clientBuilderConfiguration)
                .region(region)
                .build();
                
        // Initialize the variable that stores the total object size. 
        long totalSize = 0L;

        try {
            ObjectListing objectListing = null;
            do {
                // List objects. 
                ListObjectsRequest request = new ListObjectsRequest(bucketName).withDelimiter("/").withPrefix(keyPrefix);
                if (objectListing != null) {
                    request.setMarker(objectListing.getNextMarker());
                }
                objectListing = ossClient.listObjects(request);
                List<String> folders = objectListing.getCommonPrefixes();
                for (String folder : folders) {
                    long folderSize = calculateFolderLength(ossClient, bucketName, folder);
                    totalSize += folderSize;
                    System.out.println(folder + " : " + (folderSize / 1024) + "KB");
                }
                List<OSSObjectSummary> sums = objectListing.getObjectSummaries();
                for (OSSObjectSummary s : sums) {
                    totalSize += s.getSize();
                    System.out.println(s.getKey() + " : " + (s.getSize() / 1024) + "KB");
                }
            } while (objectListing.isTruncated());
        
            System.out.println("Total Size: " + (totalSize / 1024) + "KB");
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }

    // Calculate the total storage size of objects with the specified key prefix. 
    private static long calculateFolderLength(OSS ossClient, String bucketName, String folder) {
        long size = 0L;
        ObjectListing objectListing = null;
        do {
            // The default value of MaxKeys is 100. The maximum value of MaxKeys is 1000. 
            ListObjectsRequest request = new ListObjectsRequest(bucketName).withPrefix(folder).withMaxKeys(1000);
            if (objectListing != null) {
                request.setMarker(objectListing.getNextMarker());
            }
            objectListing = ossClient.listObjects(request);
            List<OSSObjectSummary> sums = objectListing.getObjectSummaries();
            for (OSSObjectSummary s : sums) {
                size += s.getSize();
            }
        } while (objectListing.isTruncated());
        return size;
    }
}

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
# Specify the name of the bucket. Example: examplebucket. 
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'examplebucket')
# Specify the prefix. 
prefix = 'bill/'

def calculate_storage_size(prefix):
    total_size = 0

    # List all objects whose names start with the specified prefix.
    for obj in oss2.ObjectIterator(bucket, prefix=prefix):
        total_size += obj.size  # Cumulatively add the size of each object to the total_size variable. 

    return total_size

if __name__ == "__main__":
    size = calculate_storage_size(prefix)
    print(f'Total size of objects with prefix "{prefix}": {size} bytes')

References

• FAQ about bucket inventories

• FAQ about object listing