Client-side encryption using OSS SDK for Go 2.0

Usage notes

The sample code in this topic uses the region ID cn-hangzhou of the China (Hangzhou) region and the public endpoint by default. To access resources from other Alibaba Cloud services in the same region as the bucket, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and Endpoints.
In this topic, access credentials are obtained from environment variables. For more information, see Configure access credentials (Go SDK V1).
When you use client-side encryption, you must ensure the integrity and validity of the CMK.
When you copy or migrate encrypted data, you are responsible for the integrity and validity of object metadata.

Syntax

OSS SDK for Go V2 supports two types of CMKs for client-side encryption.

RSA-based CMKs managed by yourself

The SDK provides the default Rivest-Shamir-Adleman (RSA) implementation. If you provide a CMK, you must pass the public key and private key to the SDK.
Custom CMKs

You can implement custom CMK-based encryption and decryption. This topic provides examples by using Key Management Service (KMS) V3.0.

These encryption methods protect your data on the client side. Even if your data is leaked, it cannot be decrypted by others.

Important

For more information about client-side encryption, see Client-side encryption.

To use client-side encryption, create an EncryptionClient instance and call its API operations. Objects are automatically encrypted and decrypted as part of each request.

type EncryptionClient struct {
  ...
}

func NewEncryptionClient(c *Client, masterCipher crypto.MasterCipher, optFns ...func(*EncryptionClientOptions)) (eclient *EncryptionClient, err error)

Request parameters

Parameter	Type	Description
c	*Client	An instance of a non-encryption client.
masterCipher	crypto.MasterCipher	The CMK instance for encrypting and decrypting data keys.
optFns	...func(*EncryptionClientOptions)	The configuration options of the client for client-side encryption.

EncryptionClientOptions

Parameter	Type	Description
MasterCiphers	[]crypto.MasterCipher	The CMK instances used to decrypt data keys.

Response parameters

Parameter	Type	Description
eclient	*EncryptionClient	The client instance for client-side encryption. Returned when err is nil.
err	error	The error that occurred during client creation. A nil value indicates success.

Operations supported by EncryptionClient

Basic operation	Description
GetObjectMeta	Queries standard HTTP metadata headers of the object.
HeadObject	Queries all metadata information of the object.
GetObject	Downloads and decrypts the object.
PutObject	Uploads and decrypts the object.
InitiateMultipartUpload	Initiates a multipart upload and the multipart encryption context (EncryptionMultiPartContext).
UploadPart	Uploads a part with automatic encryption. You must set the multipart encryption context when calling this operation.
CompleteMultipartUpload	Combines all parts into a complete object after all parts are uploaded.
AbortMultipartUpload	Cancels a multipart upload task and deletes all parts uploaded by the multipart upload task.
ListParts	Lists all parts that were uploaded by the multipart upload task.
Advanced operation	Description
NewDownloader	Creates a downloader instance.
NewUploader	Creates an uploader instance.
OpenFile	Creates a ReadOnlyFile instance.
Auxiliary operation	Description
Unwrap	Returns the underlying non-encryption client instance for basic operations.

Use RSA CMKs for client-side encryption

Use RSA-based CMKs to encrypt objects in simple uploads and decrypt objects in simple downloads

The following sample code uses an RSA-based CMK to encrypt objects in simple uploads and decrypt objects in simple downloads:

package main

import (
	"context"
	"flag"
	"log"
	"strings"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/crypto"
)

// Declare global variables.
var (
	region     string // The region in which the bucket is located.
	bucketName string // The name of the bucket.
	objectName string // The name of the object.
)

// Define the init function used to initialize parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	// Parse parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	// Load the default configurations and specify the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSSClient instance.
	client := oss.NewClient(cfg)

	// Specify a description for the CMK. The description cannot be changed after it is specified. You can specify only one description for a CMK. 
	// If all objects use the same CMK, the description of the CMK can be empty. However, you cannot change the CMK. 
	// If you do not specify a description for the CMK, the client cannot determine which CMK to use for decryption. 
	// We recommend that you specify a description for each CMK. The client saves the mappings between the CMKs and the descriptions. 
	materialDesc := make(map[string]string)
	materialDesc["desc"] = "your master encrypt key material describe information"

	// Create a client that contains only a CMK for client-side encryption.
	// If no download operation is performed, the private key can be set to "".
	mc, err := crypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("failed to create master rsa %v", err)

	}

	// Create a client for encryption.
	eclient, err := oss.NewEncryptionClient(client, mc)
	if err != nil {
		log.Fatalf("failed to create encryption client %v", err)
	}

	// Create a request to upload a local file by using simple upload.
	putObjRequest := &oss.PutObjectRequest{
		Bucket: oss.Ptr(bucketName),
		Key:    oss.Ptr(objectName),
		Body:   strings.NewReader("hi, simple put object"),
	}

	// Use the encryption client to upload the object.
	putObjRequestResult, err := eclient.PutObject(context.TODO(), putObjRequest)
	if err != nil {
		log.Fatalf("failed to put object with encryption client %v", err)
	}
	log.Printf("put object with encryption client result:%#v\n", putObjRequestResult)

	// Create a request to download the object by using simple download.
	getObjRequest := &oss.GetObjectRequest{
		Bucket: oss.Ptr(bucketName),
		Key:    oss.Ptr(objectName),
	}

	// Use the encryption client to download the object.
	getObjRequestResult, err := eclient.GetObject(context.TODO(), getObjRequest)
	if err != nil {
		log.Fatalf("failed to put object with encryption client %v", err)
	}
	log.Printf("put object with encryption client result:%#v\n", getObjRequestResult)

}

Use RSA-based CMKs to encrypt objects to be uploaded by using multipart upload

The following code uses an RSA-based CMK to encrypt objects in multipart upload:

package main

import (
	"bufio"
	"context"
	"flag"
	"io"
	"log"
	"math/rand"
	"sort"
	"strings"
	"sync"
	"time"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/crypto"
)

// Declare global variables.
var (
	region     string                                                                     // The region in which the bucket is located.
	bucketName string                                                                     // The name of the bucket.
	objectName string                                                                     // The name of the object.
	letters    = []rune("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ") // The character set used to generate a random string.
)

// Define the init function used to initialize parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	// Parse parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	// Load the default configurations and specify the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSSClient instance.
	client := oss.NewClient(cfg)

	// Specify a description for the CMK. The description cannot be changed after it is specified. You can specify only one description for a CMK. 
	// If all objects use the same CMK, the description of the CMK can be empty. However, you cannot change the CMK. 
	// If you do not specify a description for the CMK, the client cannot determine which CMK to use for decryption. 
	// We recommend that you specify a description for each CMK. The client saves the mappings between the CMKs and the descriptions. 
	materialDesc := make(map[string]string)
	materialDesc["desc"] = "your master encrypt key material describe information"

	// Create a client that contains only a CMK for client-side encryption.
	// If no download operation is performed, the private key can be set to "".
	mc, err := crypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("failed to create master rsa %v", err)

	}

	// Create a client for encryption.
	eclient, err := oss.NewEncryptionClient(client, mc)
	if err != nil {
		log.Fatalf("failed to create encryption client %v", err)
	}

	// Create a request to initiate the multipart upload task.
	initRequest := &oss.InitiateMultipartUploadRequest{
		Bucket: oss.Ptr(bucketName),
		Key:    oss.Ptr(objectName),
	}
	initResult, err := eclient.InitiateMultipartUpload(context.TODO(), initRequest)
	if err != nil {
		log.Fatalf("failed to initiate multi part upload %v", err)
	}

	var wg sync.WaitGroup
	var parts oss.UploadParts
	count := 3
	body := randStr(400000)
	reader := strings.NewReader(body)
	bufReader := bufio.NewReader(reader)
	content, _ := io.ReadAll(bufReader)
	partSize := len(body) / count
	var mu sync.Mutex

	for i := 0; i < count; i++ {
		wg.Add(1)
		go func(partNumber int, partSize int, i int) {
			defer wg.Done()
			partRequest := &oss.UploadPartRequest{
				Bucket:              oss.Ptr(bucketName),                                             // The name of the bucket.
				Key:                 oss.Ptr(objectName),                                             // The name of the object.
				PartNumber:          int32(partNumber),                                               // The part number.
				UploadId:            oss.Ptr(*initResult.UploadId),                                   // The upload ID.
				Body:                strings.NewReader(string(content[i*partSize : (i+1)*partSize])), // The part content.
				CSEMultiPartContext: initResult.CSEMultiPartContext,                                  // The multipart context.
			}
			partResult, err := eclient.UploadPart(context.TODO(), partRequest)
			if err != nil {
				log.Fatalf("failed to upload part %d: %v", partNumber, err)
			}
			part := oss.UploadPart{
				PartNumber: partRequest.PartNumber, // The part number.
				ETag:       partResult.ETag,        // ETag
			}
			mu.Lock()
			parts = append(parts, part)
			mu.Unlock()
		}(i+1, partSize, i)
	}
	wg.Wait()
	sort.Sort(parts)

	request := &oss.CompleteMultipartUploadRequest{
		Bucket:   oss.Ptr(bucketName),           // The name of the bucket.
		Key:      oss.Ptr(objectName),           // The name of the object.
		UploadId: oss.Ptr(*initResult.UploadId), // The upload ID.
		CompleteMultipartUpload: &oss.CompleteMultipartUpload{
			Parts: parts, // The parts uploaded.
		},
	}
	result, err := eclient.CompleteMultipartUpload(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to complete multipart upload %v", err)
	}
	log.Printf("complete multipart upload result:%#v\n", result)
}

// Generate a random string.
func randStr(n int) string {
	b := make([]rune, n)
	randMarker := rand.New(rand.NewSource(time.Now().UnixNano()))
	for i := range b {
		b[i] = letters[randMarker.Intn(len(letters))]
	}
	return string(b)
}

Use custom CMKs

Use custom CMKs to encrypt objects in simple uploads and decrypt objects in simple downloads

The SDK provides a default RSA implementation. If it does not meet your needs, you can implement encryption and decryption with custom CMKs. The following sample code uses a custom CMK in KMS V3.0 to encrypt an object in simple uploads and decrypt the object in simple downloads:

package main

import (
	"context"
	"encoding/base64"
	"encoding/json"
	"flag"
	"fmt"
	"io"
	"log"
	"strings"

	kms "github.com/aliyun/alibaba-cloud-sdk-go/services/kms"
	kmssdk "github.com/aliyun/alibabacloud-dkms-transfer-go-sdk/sdk"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
	osscrypto "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/crypto"
)

// The CreateMasterAliKms3 function creates a CMK interface implemented in KMS V3.0.
// matDesc will be converted to a JSON string.
func CreateMasterAliKms3(matDesc map[string]string, kmsID string, kmsClient *kmssdk.KmsTransferClient) (osscrypto.MasterCipher, error) {
	var masterCipher MasterAliKms3Cipher
	if kmsID == "" || kmsClient == nil {
		return masterCipher, fmt.Errorf("kmsID is empty or kmsClient is nil")
	}

	var jsonDesc string
	if len(matDesc) > 0 {
		b, err := json.Marshal(matDesc)
		if err != nil {
			return masterCipher, err
		}
		jsonDesc = string(b)
	}

	masterCipher.MatDesc = jsonDesc
	masterCipher.KmsID = kmsID
	masterCipher.KmsClient = kmsClient
	return masterCipher, nil
}

// The CMK interface in KMS.
type MasterAliKms3Cipher struct {
	MatDesc   string                    // The CMK description.
	KmsID     string                    // The CMK ID.
	KmsClient *kmssdk.KmsTransferClient // The KMS client.
}

// Obtain the wrapping algorithm of the CMK.
func (mrc MasterAliKms3Cipher) GetWrapAlgorithm() string {
	return "KMS/ALICLOUD"
}

// Obtain the CMK description.
func (mkms MasterAliKms3Cipher) GetMatDesc() string {
	return mkms.MatDesc
}

// Encrypt the data, mainly the symmetric key and initialization vector (IV) of the object.
func (mkms MasterAliKms3Cipher) Encrypt(plainData []byte) ([]byte, error) {
	base64Plain := base64.StdEncoding.EncodeToString(plainData)
	request := kms.CreateEncryptRequest()
	request.RpcRequest.Scheme = "https"
	request.RpcRequest.Method = "POST"
	request.RpcRequest.AcceptFormat = "json"

	request.KeyId = mkms.KmsID
	request.Plaintext = base64Plain

	response, err := mkms.KmsClient.Encrypt(request)
	if err != nil {
		return nil, err
	}
	return base64.StdEncoding.DecodeString(response.CiphertextBlob)
}

// Decrypt the data, mainly the symmetric key and initialization vector (IV) of the object.
func (mkms MasterAliKms3Cipher) Decrypt(cryptoData []byte) ([]byte, error) {
	base64Crypto := base64.StdEncoding.EncodeToString(cryptoData)
	request := kms.CreateDecryptRequest()
	request.RpcRequest.Scheme = "https"
	request.RpcRequest.Method = "POST"
	request.RpcRequest.AcceptFormat = "json"
	request.CiphertextBlob = string(base64Crypto)
	response, err := mkms.KmsClient.Decrypt(request)
	if err != nil {
		return nil, err
	}
	return base64.StdEncoding.DecodeString(response.Plaintext)
}

var (
	region     string // The region in which the bucket is located.
	bucketName string // The name of the bucket.
	objectName string // The name of the object.
)

// Define the init function used to initialize parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	// Parse parameters.
	flag.Parse()

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	// Load the default configurations and specify the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create the OSSClient instance.
	client := oss.NewClient(cfg)

	// Create a KMS client.
	kmsRegion := "cn-hangzhou"                // The region where the KMS instance.
	kmsAccessKeyId := "access key id"         // The AccessKey ID used to access KMS.
	kmsAccessKeySecret := "access key secret" // The AccessKey secret used to access KMS.
	kmsKeyId := "kms id" // The CMK ID.

	kmsClient, err := kmssdk.NewClientWithAccessKey(kmsRegion, kmsAccessKeyId, kmsAccessKeySecret, nil)
	if err != nil {
		log.Fatalf("failed to create kms sdk client %v", err)
	}

	// Create a description for the CMK.
	materialDesc := make(map[string]string)
	materialDesc["desc"] = "your kms encrypt key material describe information"

	// Create a CMK instance.
	masterKmsCipher, err := CreateMasterAliKms3(materialDesc, kmsKeyId, kmsClient)
	if err != nil {
		log.Fatalf("failed to create master AliKms3 %v", err)
	}

	// Create a client for encryption.
	eclient, err := oss.NewEncryptionClient(client, masterKmsCipher)
	if err != nil {
		log.Fatalf("failed to create encryption client %v", err)
	}

	// Create a request to upload an object.
	request := &oss.PutObjectRequest{
		Bucket: oss.Ptr(bucketName),         // The name of the bucket.
		Key:    oss.Ptr(objectName),         // The name of the object.
		Body:   strings.NewReader("hi kms"), // The data to be uploaded.
	}

	// Upload the object.
	result, err := eclient.PutObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to put object with encryption client %v", err)
	}
	log.Printf("put object with encryption client result:%#v\n", result)

	// Create a request to download the object.
	getRequest := &oss.GetObjectRequest{
		Bucket: oss.Ptr(bucketName), // The name of the bucket.
		Key:    oss.Ptr(objectName), // The name of the object.
	}

	// Download the object.
	getResult, err := eclient.GetObject(context.TODO(), getRequest)
	if err != nil {
		log.Fatalf("failed to get object with encryption client %v", err)
	}
	defer getResult.Body.Close()

	// Read the downloaded data.
	data, err := io.ReadAll(getResult.Body)
	if err != nil {
		log.Fatalf("failed to read all %v", err)
	}
	log.Printf("get object data:%s\n", data)
}

References

For more information about client-side encryption, see Client-side encryption.
For more information about how to implement client-side encryption by using OSS SDK for Go, see Developer Guide.
For the complete sample code for simple uploads and downloads based on RSA-based CMKs, visit GitHub.
For the complete sample code for simple uploads and downloads based on KMS CMKs, visit GitHub.