Access control and log management for vector buckets - Object Storage Service

Vector buckets offer access control and log management to ensure data security, compliance, and observability.

Access control

Vector buckets support bucket policy and RAM policy.

Bucket policy: A resource-based authorization policy attached to a bucket to grant other Alibaba Cloud accounts, RAM users, or anonymous users access to specified vector resources.
RAM policy: An identity-based authorization policy attached to a RAM user, user group, or role to define which vector bucket resources they can access.

Supported actions

API	Action	Description
PutVectorBucket	oss:PutVectorBucket	Creates a vector bucket.
GetVectorBucket	oss:GetVectorBucket	Gets the details of a vector bucket.
ListVectorBuckets	oss:ListVectorBuckets	Lists all vector buckets owned by the requester.
DeleteVectorBucket	oss:DeleteVectorBucket	Deletes a vector bucket.
PutBucketLogging	oss:PutBucketLogging	Enables logging for a vector bucket.
PutBucketLogging	oss:PutObject	Required for writing logs from the source vector bucket to a target bucket when logging is enabled.
GetBucketLogging	oss:GetBucketLogging	Gets the logging configuration of a vector bucket.
DeleteBucketLogging	oss:DeleteBucketLogging	Disables logging for a vector bucket.
PutBucketPolicy	oss:PutBucketPolicy	Sets the authorization policy for a specified vector bucket.
GetBucketPolicy	oss:GetBucketPolicy	Gets the authorization policy of a specified vector bucket.
DeleteBucketPolicy	oss:DeleteBucketPolicy	Deletes the authorization policy of a specified vector bucket.
PutVectorIndex	oss:PutVectorIndex	Creates a vector index.
GetVectorIndex	oss:GetVectorIndex	Gets the details of a vector index.
ListVectorIndexes	oss:ListVectorIndexes	Lists all vector indexes in a vector bucket.
DeleteVectorIndex	oss:DeleteVectorIndex	Deletes a vector index.
PutVectors	oss:PutVectors	Writes vector data.
GetVectors	oss:GetVectors	Gets specified vector data.
ListVectors	oss:ListVectors	Lists all vector data in a vector index.
QueryVectors	oss:QueryVectors	Performs a vector similarity search.
DeleteVectors	oss:DeleteVectors	Deletes specified vector data from a vector index.

Resource format

Resource level	Format	Example
All vector resources	`acs:ossvector:::` `acs:ossvector:::/*`	`acs:ossvector:::` `acs:ossvector:::/*`
vector bucket	`acs:ossvector:{region}:{account_id}:{bucket_name}`	`acs:ossvector:::my-vector-bucket`
vector index	`acs:ossvector:{region}:{account_id}:{bucket_name}/{index_name}`	`acs:ossvector:::my-vector-bucket/my-index`

Bucket policy

Use a bucket policy to grant RAM users and other accounts access to specified vector resources.

Console

On the Vector Buckets page, click the target bucket. In the left-side navigation pane, choose Access Control > Bucket Authorization Policy.

Click Add By Syntax Policy and enter the policy in the policy editor. For example, to grant the user with UID 11********80**** permissions to read and write vector data in the my-index index of the my-vector-bucket bucket:

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "oss:PutVectors",
        "oss:GetVectors"
      ],
      "Principal": [
         "1142323451******"
      ],   
      "Resource": [
        "acs:ossvector:*:*:my-vector-bucket/my-index"
      ]
    }
  ]
}

Click OK.

ossutil

The following example shows how to set a bucket policy using a JSON configuration file named vector-policy.json. The file contains the following content:

{
   "Version":"1",
   "Statement":[
       {
         "Action":[
           "oss:PutVectors",
           "oss:GetVectors"
        ],
        "Effect":"Deny",
        "Principal":["1234567890"],
        "Resource":["acs:ossvector:cn-hangzhou:1234567890:*"]
       }
    ]
}

ossutil vectors-api put-bucket-policy --bucket vector-example --body file://vector-policy.json

You can set a bucket policy for a vector bucket using a JSON configuration parameter:

ossutil vectors-api put-bucket-policy --bucket vector-example --body "{\"Version\":\"1\",\"Statement\":[{\"Action\":[\"oss:PutVectors\",\"oss:GetVectors\",\"oss:QueryVectors\"],\"Effect\":\"Allow\",\"Principal\":[\"1234567890\"],\"Resource\":[\"acs:ossvector:cn-hangzhou:1234567890:bucket/vector-example/*\"]}]}"

SDK

Python

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.vectors as oss_vectors

parser = argparse.ArgumentParser(description="vector put bucket policy sample")
parser.add_argument('--region', help='The region in which the bucket is located.', required=True)
parser.add_argument('--bucket', help='The name of the bucket.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS')
parser.add_argument('--account_id', help='The account id.', required=True)

def main():
    args = parser.parse_args()

    # Loading credentials values from the environment variables
    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    # Using the SDK's default configuration
    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    cfg.account_id = args.account_id
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    vector_client = oss_vectors.Client(cfg)


    policy_content = '''
        {
           "Version":"1",
           "Statement":[
               {
                 "Action":[
                   "oss:PutVectors",
                   "oss:GetVectors"
                ],
                "Effect":"Deny",
                "Principal":["1234567890"],
                "Resource":["acs:ossvector:cn-hangzhou:1234567890:*"]
               }
            ]
         }
    '''

    result = vector_client.put_bucket_policy(oss_vectors.models.PutBucketPolicyRequest(
        bucket=args.bucket,
        body=policy_content
    ))

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
    )

if __name__ == "__main__":
    main()

Go

package main

import (
	"context"
	"flag"
	"log"
	"strings"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/vectors"
)

var (
	region     string
	bucketName string
	accountId  string
)

func init() {
	flag.StringVar(&region, "region", "", "The region in which the vector bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the vector bucket.")
	flag.StringVar(&accountId, "account-id", "", "The ID of the vector account.")
}

func main() {
	flag.Parse()
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	if len(accountId) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, accountId required")
	}

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region).WithAccountId(accountId)

	client := vectors.NewVectorsClient(cfg)

	request := &vectors.PutBucketPolicyRequest{
		Bucket: oss.Ptr(bucketName),
		Body: strings.NewReader(`{
			   "Version":"1",
			   "Statement":[
			   {
				 "Action":[
				   "oss:PutVectors",
				   "oss:GetVectors"
				],
				"Effect":"Deny",
				"Principal":["1234567890"],
				"Resource":["acs:ossvector:cn-hangzhou:1234567890:*"]
			   }
			  ]
			 }`),
	}
	result, err := client.PutBucketPolicy(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to put vector bucket policy %v", err)
	}

	log.Printf("put vector bucket policy result:%#v\n", result)
}

API

Call the PutBucketPolicy operation to set a bucket policy for a vector bucket.

RAM policy

You can use RAM policies to configure permissions for vector buckets for RAM users or roles in the RAM console. The policies support resource granularity at the Index level.

Scenario 1: Grant a RAM user full permissions on a specific vector index

{                                                                           
   "Version": "1",                                                             
   "Statement": [                                                              
     {                                                                         
       "Effect": "Allow",                                                      
       "Action": "*",                                                      
       "Resource": [                                                           
         "acs:ossvector:*:*:my-vector-bucket/my-index"                         
       ]                                                                       
     }                                                                         
   ]                                                                           
 }

Scenario 2: Grant a RAM user full control over a vector bucket

  {                                                                             
    "Version": "1",                                                             
    "Statement": [                                                              
      {                                                                         
        "Effect": "Allow",                                                      
        "Action": "*",                                                      
        "Resource": [                                                           
          "acs:ossvector:*:*:my-vector-bucket",                                 
          "acs:ossvector:*:*:my-vector-bucket/*"                                
        ]                                                                       
      }                                                                         
    ]                                                                           
  }

Log management

Enable access logging to store access records in a specified OSS bucket. Use these logs for security audits, performance analysis, and troubleshooting.

Console

On the Vector Buckets page, click the target bucket. In the left-side navigation pane, choose Log management > Log storage.
Turn on the Log storage switch and configure the following parameters:
- Target Storage Location: Select the bucket to store the log files. This bucket must be in the same region as the vector bucket.
- Log Prefix: Set the directory and prefix for the log files, such as MyLog-.
- Authorization Role: Use the default logging role AliyunOSSLoggingDefaultRole or select a custom role.

ossutil

The following examples show how to enable log storage for a bucket named examplebucket. The log file prefix is MyLog-, and the access logs are stored in the examplebucket bucket.

You can use a JSON configuration file. The bucket-logging-status.json file contains the following content:

{
  "BucketLoggingStatus": {
    "LoggingEnabled": {
      "TargetBucket": "examplebucket",
      "TargetPrefix": "MyLog-",
      "LoggingRole": "AliyunOSSLoggingDefaultRole"
    }
  }
}

Example command:

ossutil vectors-api put-bucket-logging --bucket examplebucket --bucket-logging-status file://bucket-logging-status.json

You can use JSON configuration parameters. Example command:

ossutil vectors-api put-bucket-logging --bucket examplebucket --bucket-logging-status "{\"BucketLoggingStatus\":{\"LoggingEnabled\":{\"TargetBucket\":\"examplebucket\",\"TargetPrefix\":\"MyLog-\",\"LoggingRole\":\"AliyunOSSLoggingDefaultRole\"}}}"

SDK

Python

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.vectors as oss_vectors

parser = argparse.ArgumentParser(description="Sample for enabling log storage for a vector bucket")
parser.add_argument('--region', help='The region where the bucket is located.', required=True)
parser.add_argument('--bucket', help='The bucket name.', required=True)
parser.add_argument('--endpoint', help='The endpoint to access OSS.')
parser.add_argument('--account_id', help='The account ID.', required=True)
parser.add_argument('--target_bucket', help='The destination bucket for storing logs.', required=True)

def main():
    args = parser.parse_args()

    # Loading credentials values from the environment variables
    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    # Using the SDK's default configuration
    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    cfg.account_id = args.account_id
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    vector_client = oss_vectors.Client(cfg)

    result = vector_client.put_bucket_logging(oss_vectors.models.PutBucketLoggingRequest(
        bucket=args.bucket,
        bucket_logging_status=oss_vectors.models.BucketLoggingStatus(
            logging_enabled=oss_vectors.models.LoggingEnabled(
                target_bucket=args.target_bucket,
                target_prefix='log-prefix',
                logging_role='AliyunOSSLoggingDefaultRole'
            )
        )
    ))

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
    )

if __name__ == "__main__":
    main()

Go

package main

import (
	"context"
	"flag"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/vectors"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

var (
	region     string
	bucketName string
	accountId  string
)

func init() {
	flag.StringVar(&region, "region", "", "The region in which the vector bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the vector bucket.")
	flag.StringVar(&accountId, "account-id", "", "The ID of the vector account.")
}

func main() {
	flag.Parse()
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	if len(accountId) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, accountId required")
	}

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region).WithAccountId(accountId)

	client := vectors.NewVectorsClient(cfg)

	request := &vectors.PutBucketLoggingRequest{
		Bucket: oss.Ptr(bucketName),
		BucketLoggingStatus: &vectors.BucketLoggingStatus{
			&vectors.LoggingEnabled{
				TargetBucket: oss.Ptr("TargetBucket"),
				TargetPrefix: oss.Ptr("TargetPrefix"),
				LoggingRole:  oss.Ptr("AliyunOSSLoggingDefaultRole"),
			},
		},
	}
	result, err := client.PutBucketLogging(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to put vector bucket logging %v", err)
	}

	log.Printf("put vector bucket logging result:%#v\n", result)
}

API

Call the PutBucketLogging operation to enable log storage for a vector bucket.

Log file naming convention

The naming convention for stored log files is as follows:

<TargetPrefix><SourceBucket>YYYY-mm-DD-HH-MM-SS-UniqueString

Field	Description
TargetPrefix	The prefix of the log file name.
SourceBucket	The name of the source bucket that generates access logs.
YYYY-mm-DD-HH-MM-SS	The time partition of the log. From left to right, they represent year, month, day, hour, minute, and second. The stored logs are organized by hour. For example, if HH is 01, the log file contains log information from 01:00:00 to 01:59:59. MM and SS are both pushed as 00.
UniqueString	A system-generated string that uniquely identifies the log file.

Log format and examples

Log format

OSS access logs contain information about the requester and the accessed resource. The format is as follows:

RemoteIP Reserved Reserved Time "RequestURL" HTTPStatus SentBytes RequestTime "Referer" "UserAgent" "HostName" "RequestID" "LoggingFlag" "RequesterAliyunID" "Operation" "BucketName" "ObjectName" ObjectSize ServerCostTime "ErrorCode" RequestLength "UserID" DeltaDataSize "SyncRequest" "StorageClass" "TargetStorageClass" "TransmissionAccelerationAccessPoint" "AccessKeyID" "BucketARN"

Field	Example	Description
RemoteIP	192.168.0.1	The IP address of the requester.
Reserved	-	Reserved field. The value is fixed to -.
Reserved	-	Reserved field. The value is fixed to -.
Time	03/Jan/2021:14:59:49 +0800	The time when OSS received the request.
RequestURL	GET /example.jpg HTTP/1.0	The request URL that contains the query string. OSS ignores query string parameters that start with `x-`, but these parameters are recorded in the access log. Use a query string parameter that starts with `x-` to mark a request and then use this mark to quickly find the corresponding log.
HTTPStatus	200	The HTTP status code returned by OSS.
SentBytes	999131	The downstream traffic generated by the request, in bytes.
RequestTime	127	The time taken to complete the request, in ms.
Referer	http://www.aliyun.com/product/oss	The HTTP Referer of the request.
UserAgent	curl/7.15.5	The User-Agent header of the HTTP request.
HostName	examplebucket.oss-cn-hangzhou.aliyuncs.com	The destination domain name accessed by the request.
RequestID	5FF16B65F05BC932307A3C3C	The request ID.
LoggingFlag	true	Indicates whether logging is enabled. Valid values: true: enabled. false: not enabled.
RequesterAliyunID	16571836914537****	The user ID of the requester. A value of - indicates an anonymous access.
Operation	GetObject	The request type.
BucketName	examplebucket	The name of the destination bucket.
ObjectName	example.jpg	The name of the destination object.
ObjectSize	999131	The size of the destination object, in bytes.
ServerCostTime	88	The time taken by OSS to process the request, in milliseconds.
ErrorCode	-	The error code returned by OSS. A value of - indicates that no error code was returned.
RequestLength	302	The length of the request, in bytes.
UserID	16571836914537****	The ID of the bucket owner.
DeltaDataSize	-	The change in the object size. A value of - indicates that the request does not involve an object write operation.
SyncRequest	-	The request type. Valid values: -: A general request. cdn: A CDN origin request. lifecycle: A request to dump or delete data made through a lifecycle rule.
StorageClass	Standard	The storage class of the destination object. Valid values: Standard: Standard. IA: Infrequent Access. Archive: Archive Storage. Cold Archive: Cold Archive. DeepCold Archive: Deep Cold Archive. -: The object storage class was not obtained.
TargetStorageClass	-	Indicates whether the storage class of the object was changed by a lifecycle rule or a CopyObject operation. Valid values: Standard: The storage class is changed to Standard. IA: The storage class is changed to Infrequent Access. Archive: The storage class is changed to Archive Storage. Cold Archive: The storage class is changed to Cold Archive. DeepCold Archive: The storage class is changed to Deep Cold Archive. -: No storage class conversion operation is involved.
TransmissionAccelerationAccessPoint	-	The acceleration endpoint used when accessing the destination bucket through a transfer acceleration domain name. For example, if a requester accesses the destination bucket through an endpoint in China (Hangzhou), the value is cn-hangzhou. A value of - indicates that a transfer acceleration domain name was not used or the acceleration endpoint is in the same region as the destination bucket.
AccessKeyID	LTAI****************	The AccessKey ID of the requester. If the request is initiated from the console, a temporary AccessKey ID that starts with TMP is displayed in the log field. If the request is initiated from a tool or an SDK using a long-term key, a common AccessKey ID is displayed in the log field. Example: `LTAI**************`. If the request is initiated using temporary access credentials from Security Token Service (STS), a temporary AccessKey ID that starts with STS is displayed. Note** The AccessKey ID field displays - for anonymous requests.
BucketArn	acs:oss***************	The globally unique resource descriptor of the bucket.

Log example

192.168.0.1 - - [03/Jan/2021:14:59:49 +0800] "GET /example.jpg HTTP/1.0" 200 999131 127 "http://www.aliyun.com/product/oss" "curl/7.15.5" "examplebucket.oss-cn-hangzhou.aliyuncs.com" "5FF16B65F05BC932307A3C3C" "true" "16571836914537****" "GetObject" "examplebucket" "example.jpg" 999131 88 "-" 302 "16571836914537****" - "cdn" "standard" "-" "-" "LTAI****************" "acs:oss***************"

After log files are stored in the specified OSS bucket, you can use Simple Log Service to analyze them. Before you can analyze the log files, you must import them to Simple Log Service. For more information about how to import data, see Import OSS data. For more information about the analysis features of Simple Log Service, see Query and analysis overview.