MetaSearch in Object Storage Service (OSS) is a search and retrieval feature that leverages data indexing to enable rapid object retrieval based on specific metadata criteria, such as object name, ETag, storage class, size, and last modified time.
Notes
-
Only OSS SDK for Java 3.16.0 and later support MetaSearch.
-
MetaSearch is supported only for buckets located in the China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Shenzhen), China (Guangzhou), China (Chengdu), China (Hong Kong), Singapore, Indonesia (Jakarta), Germany (Frankfurt), US (Virginia), US (Silicon Valley), and UK (London) regions.
-
In this topic, the public endpoint of the China (Hangzhou) region is used. To access OSS from other Alibaba Cloud services in the same region, use an internal endpoint. For details about supported regions and endpoints, see Regions and endpoints.
-
In this topic, access credentials are obtained from environment variables. For more information, see Configure access credentials.
-
This topic demonstrates creating an OSSClient instance with an OSS endpoint. For alternative configurations, such as using a custom domain or authenticating with credentials from Security Token Service (STS), see Client configuration.
-
By default, an Alibaba Cloud account has permissions to perform data indexing operations. If you want to use a RAM user or STS to perform data indexing operations, make sure you have the required permissions:
-
To enable the metadata management feature, you must have the
oss:OpenMetaQuerypermission. -
To query information about a metadata index library, you must have the
oss:GetMetaQueryStatuspermission. -
To query objects that meet specific conditions, you must have the
oss:DoMetaQuerypermission. -
To disable the metadata management feature, you must have the
oss:CloseMetaQuerypermission.
-
Enable the MetaSearch feature
After you enable the MetaSearch feature for a bucket, OSS creates a metadata index library for the bucket and creates metadata indexes for all objects in the bucket. After the metadata index library is created, OSS continues to perform quasi-real-time scans for incremental objects in the bucket and creates metadata indexes for the incremental objects. The following sample code enables the MetaSearch feature for a bucket:
import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.common.comm.SignVersion;
import com.aliyun.oss.model.MetaQueryMode;
public class OpenMetaQuery {
public static void main(String[] args) throws com.aliyuncs.exceptions.ClientException {
// In this example, the endpoint of the China (Hangzhou) region is used.
String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// Specify the name of the bucket. Example: examplebucket.
String bucketName = "examplebucket";
// Obtain access credentials from environment variables.
// Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
String region = "cn-hangzhou";
// Create an OSSClient instance.
//Call the shutdown method to release associated resources when the OSSClient instance is no longer used.
ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
OSS ossClient = OSSClientBuilder.create()
.endpoint(endpoint)
.credentialsProvider(credentialsProvider)
.clientConfiguration(clientBuilderConfiguration)
.region(region)
.build();
try {
// Enable the MetaSearch feature.
ossClient.openMetaQuery(bucketName);
} catch (OSSException oe) {
System.out.println("Error Message:" + oe.getErrorMessage());
System.out.println("Error Code:" + oe.getErrorCode());
System.out.println("Request ID:" + oe.getRequestId());
System.out.println("Host ID:" + oe.getHostId());
} catch (ClientException ce) {
System.out.println("Error Message: " + ce.getMessage());
} finally {
// Shut down the OSSClient instance.
if(ossClient != null){
ossClient.shutdown();
}
}
}
}
Query information about the metadata index library of a bucket
The following sample code queries information about the metadata index library of a bucket:
import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.common.comm.SignVersion;
import com.aliyun.oss.model.GetMetaQueryStatusResult;
public class GetMetaQueryStatus {
public static void main(String[] args) throws com.aliyuncs.exceptions.ClientException {
// In this example, the endpoint of the China (Hangzhou) region is used.
String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// Specify the name of the bucket. Example: examplebucket.
String bucketName = "examplebucket";
// Obtain access credentials from environment variables.
// Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
String region = "cn-hangzhou";
// Create an OSSClient instance.
// Call the shutdown method to release associated resources when the OSSClient instance is no longer used.
ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
OSS ossClient = OSSClientBuilder.create()
.endpoint(endpoint)
.credentialsProvider(credentialsProvider)
.clientConfiguration(clientBuilderConfiguration)
.region(region)
.build();
try {
// Query information about the metadata index library of the bucket.
GetMetaQueryStatusResult getResult = ossClient.getMetaQueryStatus(bucketName);
// Query the scan type.
System.out.println(getResult.getPhase());
// Query the status of the metadata index library.
System.out.println(getResult.getState());
// Query the time when the metadata index library was created.
System.out.println(getResult.getCreateTime());
// Query the time when the metadata index library was last updated.
System.out.println(getResult.getUpdateTime());
} catch (OSSException oe) {
System.out.println("Error Message:" + oe.getErrorMessage());
System.out.println("Error Code:" + oe.getErrorCode());
System.out.println("Request ID:" + oe.getRequestId());
System.out.println("Host ID:" + oe.getHostId());
} catch (ClientException ce) {
System.out.println("Error Message: " + ce.getMessage());
} finally {
// Shut down the OSSClient instance.
if(ossClient != null){
ossClient.shutdown();
}
}
}
}
Query objects that meet specific conditions
The following sample code queries objects that meet specific conditions and returns matched objects based on the specific fields and in the specified sorting order:
import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.common.comm.SignVersion;
import com.aliyun.oss.model.*;
import java.util.ArrayList;
import java.util.List;
public class DoMetaQuery {
public static void main(String[] args) throws Exception {
// In this example, the endpoint of the China (Hangzhou) region is used.
String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// Specify the name of the bucket. Example: examplebucket.
String bucketName = "examplebucket";
// Obtain access credentials from environment variables.
// Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
String region = "cn-hangzhou";
// Create an OSSClient instance.
// Call the shutdown method to release associated resources when the OSSClient instance is no longer used.
ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
OSS ossClient = OSSClientBuilder.create()
.endpoint(endpoint)
.credentialsProvider(credentialsProvider)
.clientConfiguration(clientBuilderConfiguration)
.region(region)
.build();
try {
// Query objects that meet specific conditions and return objects based on specific fields and in the specified sorting order.
int maxResults = 20;
// Query objects that are smaller than 1,048,576 bytes in size, return up to 20 objects at a time, and sort the objects in ascending order.
String query = "{\"Field\": \"Size\",\"Value\": \"1048576\",\"Operation\": \"lt\"}";
String sort = "Size";
DoMetaQueryRequest doMetaQueryRequest = new DoMetaQueryRequest(bucketName, maxResults, query, sort);
Aggregation aggregationRequest = new Aggregation();
Aggregations aggregations = new Aggregations();
List<Aggregation> aggregationList = new ArrayList<Aggregation>();
// Specify the name of the field that is used in the aggregate operation.
aggregationRequest.setField("Size");
// Specify the operator that is used in the aggregate operation. max indicates the maximum value.
aggregationRequest.setOperation("max");
aggregationList.add(aggregationRequest);
aggregations.setAggregation(aggregationList);
// Specify the aggregate operation.
doMetaQueryRequest.setAggregations(aggregations);
doMetaQueryRequest.setOrder(SortOrder.ASC);
DoMetaQueryResult doMetaQueryResult = ossClient.doMetaQuery(doMetaQueryRequest);
if(doMetaQueryResult.getFiles() != null){
for(ObjectFile file : doMetaQueryResult.getFiles().getFile()){
System.out.println("Filename: " + file.getFilename());
// Query the ETag value that is used to identify the content of the object.
System.out.println("ETag: " + file.getETag());
// Query the access control list (ACL) of the object.
System.out.println("ObjectACL: " + file.getObjectACL());
// Query the type of the object.
System.out.println("OssObjectType: " + file.getOssObjectType());
// Query the storage class of the object.
System.out.println("OssStorageClass: " + file.getOssStorageClass());
// Query the number of tags of the object.
System.out.println("TaggingCount: " + file.getOssTaggingCount());
if(file.getOssTagging() != null){
for(Tagging tag : file.getOssTagging().getTagging()){
System.out.println("Key: " + tag.getKey());
System.out.println("Value: " + tag.getValue());
}
}
if(file.getOssUserMeta() != null){
for(UserMeta meta : file.getOssUserMeta().getUserMeta()){
System.out.println("Key: " + meta.getKey());
System.out.println("Value: " + meta.getValue());
}
}
}
} else if(doMetaQueryResult.getAggregations() != null){
for(Aggregation aggre : doMetaQueryResult.getAggregations().getAggregation()){
// Query the field by which the aggregation is performed.
System.out.println("Field: " + aggre.getField());
// Query the aggregation operator.
System.out.println("Operation: " + aggre.getOperation());
// Query the result of the aggregate operation.
System.out.println("Value: " + aggre.getValue());
if(aggre.getGroups() != null && aggre.getGroups().getGroup().size() > 0){
// Query the value of the aggregation.
System.out.println("Groups value: " + aggre.getGroups().getGroup().get(0).getValue());
// Query the total number of the aggregations.
System.out.println("Groups count: " + aggre.getGroups().getGroup().get(0).getCount());
}
}
} else {
System.out.println("NextToken: " + doMetaQueryResult.getNextToken());
}
} catch (OSSException oe) {
System.out.println("Error Message:" + oe.getErrorMessage());
System.out.println("Error Code:" + oe.getErrorCode());
System.out.println("Request ID:" + oe.getRequestId());
System.out.println("Host ID:" + oe.getHostId());
} catch (ClientException ce) {
System.out.println("Error Message: " + ce.getMessage());
} finally {
// Shut down the OSSClient instance.
ossClient.shutdown();
}
}
}
Disable the MetaSearch feature
The following sample code disables the MetaSearch feature:
import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.common.comm.SignVersion;
public class CloseMetaQuery {
public static void main(String[] args) throws Exception {
// In this example, the endpoint of the China (Hangzhou) region is used.
String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// Specify the name of the bucket. Example: examplebucket.
String bucketName = "examplebucket";
// Obtain access credentials from environment variables.
// Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
String region = "cn-hangzhou";
// Create an OSSClient instance.
// Call the shutdown method to release associated resources when the OSSClient instance is no longer used.
ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
OSS ossClient = OSSClientBuilder.create()
.endpoint(endpoint)
.credentialsProvider(credentialsProvider)
.clientConfiguration(clientBuilderConfiguration)
.region(region)
.build();
try {
// Disable the MetaSearch feature for the bucket.
ossClient.closeMetaQuery(bucketName);
} catch (OSSException oe) {
System.out.println("Error Message:" + oe.getErrorMessage());
System.out.println("Error Code:" + oe.getErrorCode());
System.out.println("Request ID:" + oe.getRequestId());
System.out.println("Host ID:" + oe.getHostId());
} catch (ClientException ce) {
System.out.println("Error Message: " + ce.getMessage());
} finally {
// Shut down the OSSClient instance.
if(ossClient != null){
ossClient.shutdown();
}
}
}
}
References
-
For more information about the API operation that enables the metadata management feature, see OpenMetaQuery.
-
For more information about the API operation that queries information about a metadata index library, see GetMetaQueryStatus.
-
For more information about the API operation that queries objects that meet specific conditions and lists objects based on specific fields and sorting order, see DoMetaQuery.
-
For more information about the API operation that disables the metadata management feature, see CloseMetaQuery.