Use Go to configure client-side encryption to encrypt data locally before uploading to OSS - Object Storage Service

When you enable client-side encryption, objects are encrypted locally before they are uploaded to Object Storage Service (OSS). Only users who have the key can decrypt the objects. This enhances data security during transmission and storage.

Disclaimer

When you use client-side encryption, you must ensure the integrity and validity of the CMK. If the CMK is incorrectly used or lost due to improper maintenance, you are responsible for all losses and consequences caused by decryption failures.
When you copy or migrate encrypted data, you are responsible for the integrity and validity of object metadata. If the encrypted metadata is incorrect or lost due to improper maintenance, you are responsible for all losses and consequences caused by data decryption failures.

Scenarios

Highly sensitive data: You may want to encrypt sensitive data, such as personally identifiable information (PII), financial transaction records, and medical or health data, before it leaves your local environment. This ensures that your raw data is protected even if it is intercepted during transmission.
Compliance requirements: Some industries and regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) and the General Data Protection Regulation (GDPR), require strict control over the encryption of data stored on third-party platforms. Client-side encryption helps you meet these compliance requirements because you manage the keys. The keys are not transmitted over the network or directly controlled by cloud service providers.
Enhanced autonomous control: You may want full control over the encryption process, including selecting encryption algorithms and managing and rotating keys. Client-side encryption helps you achieve this goal and ensures that only authorized users can decrypt and access data.
Cross-region data migration security: When you migrate data from one region to another, client-side encryption ensures that the data remains encrypted throughout the migration process. This enhances data security during transmission over the Internet.

Usage notes

In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.
In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.
In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Configure OSSClient instances.

Background information

In client-side encryption, a random data key is generated for each object to perform symmetric encryption on the object. The client uses a CMK to encrypt the random data key. The encrypted data key is uploaded as part of the object metadata and stored in the OSS server. When an encrypted object is downloaded, the client uses the CMK to decrypt the random data key, and then uses the decrypted data key to decrypt the object. To ensure data security, the CMK is used only on the client and is not transmitted over the network or stored on the server.

Important

Client-side encryption supports multipart upload for objects larger than 5 GB in size. When you use multipart upload to upload an object, you must specify the total size of the object and the size of each part. The size of each part except for the last part must be the same and be a multiple of 16.
After you upload objects encrypted on the client, object metadata related to client-side encryption is protected, and cannot be modified by calling the CopyObject operation.

Encryption methods

You can use two types of CMKs for client-side encryption:

KMS-managed CMKs
When you use a CMK managed in Key Management Service (KMS) for client-side encryption, you must provide OSS SDK for Python with the CMK ID.
RSA-based CMKs managed by yourself
When you use a CMK managed by yourself for client-side encryption, you must send the public key and the private key of your CMK to OSS SDK for Python as parameters.

You can use the preceding encryption methods to prevent data leaks and protect your data on the client. Even if your data is leaked, the data cannot be decrypted by others.

Encryption metadata

Parameter	Description	Required
x-oss-meta-client-side-encryption-key	The encrypted data key. This is a Base64-encoded string of the data key that is encrypted using a master key.	Yes
x-oss-meta-client-side-encryption-start	The random initial value used for data encryption. This is a Base64-encoded string of the initial value that is encrypted using a master key.	Yes
x-oss-meta-client-side-encryption-cek-alg	The data encryption algorithm.	Yes
x-oss-meta-client-side-encryption-wrap-alg	The data key encryption algorithm.	Yes
x-oss-meta-client-side-encryption-matdesc	The description of the master key in JSON format. Warning We strongly recommend that you configure a description for each master key and save the mapping between the master key and its description. Otherwise, you cannot change the master key for encryption.	No
x-oss-meta-client-side-encryption-unencrypted-content-length	The length of the data before encryption. This parameter is not generated if `content-length` is not specified.	No
x-oss-meta-client-side-encryption-unencrypted-content-md5	The MD5 hash of the data before encryption. This parameter is not generated if an MD5 hash is not specified.	No
x-oss-meta-client-side-encryption-data-size	If you encrypt a file using multipart upload, you must pass the total size of the file when you initialize the multipart upload.	Yes (for multipart upload)
x-oss-meta-client-side-encryption-part-size	If you encrypt a file using multipart upload, you must pass the part size when you initialize the multipart upload. Note The part size must be a multiple of 16.	Yes (for multipart upload)

Sample code

Upload and download an object using an RSA master key

The following sample code shows how to upload and download an object using an RSA master key:

package main

import (
	"bytes"
	"io"
	"log"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	osscrypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance.
	// Set yourEndpoint to the endpoint of the bucket. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. For more information about other regions, see the actual documentation.
	// Set yourRegion to the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to cn-hangzhou. For more information about other regions, see the actual documentation.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Set the signature version.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

	// Create a description for the master key. The description cannot be modified after it is created. The master key description and the master key have a one-to-one mapping.
	// If all objects use the same master key, the master key description can be empty. However, you cannot change the master key later.
	// If the master key description is empty, you cannot determine which master key is used for decryption.
	// We strongly recommend that you configure a master key description (a JSON string) for each master key and have the client save the mapping between the master key and its description. The server does not save this mapping.

	// A map converted from the master key description (a JSON string).
	materialDesc := map[string]string{
		"desc": "your master encrypt key material describe information",
	}

	// Create a master key object based on the master key description.
	// Set yourRsaPublicKey to the public key information of the master key that you manage, and set yourRsaPrivateKey to the private key information of the master key that you manage.
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("Error creating master RSA cipher: %v", err)
	}

	// Create an encryption interface based on the master key object and use AES-CTR for encryption.
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// Obtain a created bucket for client-side encryption.
	// A client-side encrypted bucket is used in a similar way to a regular bucket.
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

	// Automatic encryption during PutObject.
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		log.Fatalf("Error putting object: %v", err)
	}

	// Automatic decryption during GetObject.
	body, err := cryptoBucket.GetObject("yourObjectName")
	if err != nil {
		log.Fatalf("Error getting object: %v", err)
	}
	defer body.Close()

	data, err := io.ReadAll(body)
	if err != nil {
		log.Fatalf("Error reading object data: %v", err)
	}
	log.Printf("Data: %s", string(data))
}

Upload an object in parts using an RSA master key

The following sample code shows how to upload an object in parts using an RSA master key:

package main

import (
	"log"
	"os"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	osscrypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance.
	// Set yourEndpoint to the endpoint of the bucket. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. For more information about other regions, see the actual documentation.
	// Set yourRegion to the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to cn-hangzhou. For more information about other regions, see the actual documentation.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Set the signature version.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

	// Create a description for the master key. The description cannot be modified after it is created. The master key description and the master key have a one-to-one mapping.
	// If all objects use the same master key, the master key description can be empty. However, you cannot change the master key later.
	// If the master key description is empty, you cannot determine which master key is used for decryption.
	// We strongly recommend that you configure a master key description (a JSON string) for each master key and have the client save the mapping between the master key and its description. The server does not save this mapping.

	// A map converted from the master key description (a JSON string).
	materialDesc := map[string]string{
		"desc": "your master encrypt key material describe information",
	}

	// Create a master key object based on the master key description.
	// Set yourRsaPublicKey to the public key information of the master key that you manage, and set yourRsaPrivateKey to the private key information of the master key that you manage.
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("Error creating master RSA cipher: %v", err)
	}

	// Create an encryption interface based on the master key object and use AES-CTR for encryption.
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// Obtain a created bucket for client-side encryption.
	// A client-side encrypted bucket is used in a similar way to a regular bucket.
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

	fileName := "yourLocalFilePath"
	fileInfo, err := os.Stat(fileName)
	if err != nil {
		log.Fatalf("Error getting file info: %v", err)
	}
	fileSize := fileInfo.Size()

	// Encryption context information.
	var cryptoContext osscrypto.PartCryptoContext
	cryptoContext.DataSize = fileSize

	// The expected number of parts. The actual number of parts is based on subsequent calculations.
	expectPartCount := int64(10)

	// The part size for AES-CTR encryption must be aligned to 16 bytes.
	cryptoContext.PartSize = (fileSize / expectPartCount / 16) * 16

	// Initialize the multipart upload.
	imur, err := cryptoBucket.InitiateMultipartUpload("yourObjectName", &cryptoContext)
	if err != nil {
		log.Fatalf("Error initiating multipart upload: %v", err)
	}

	// Split the file.
	chunks, err := oss.SplitFileByPartSize(fileName, cryptoContext.PartSize)
	if err != nil {
		log.Fatalf("Error splitting file: %v", err)
	}

	var partsUpload []oss.UploadPart
	for _, chunk := range chunks {
		part, err := cryptoBucket.UploadPartFromFile(imur, fileName, chunk.Offset, chunk.Size, int(chunk.Number), cryptoContext)
		if err != nil {
			log.Fatalf("Error uploading part: %v", err)
		}
		partsUpload = append(partsUpload, part)
	}

	// Complete the multipart upload.
	_, err = cryptoBucket.CompleteMultipartUpload(imur, partsUpload)
	if err != nil {
		log.Fatalf("Error completing multipart upload: %v", err)
	}

	log.Println("Multipart upload completed successfully")
}

Decrypt objects that are encrypted with different RSA master keys

The following sample code shows how to decrypt objects that were encrypted with different RSA master keys:

package main

import (
	"bytes"
	"io"
	"log"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	osscrypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

// Query the master key based on the master key description. To decrypt objects encrypted with different master keys, provide this interface.
type MockRsaManager struct{}

func (mg *MockRsaManager) GetMasterKey(matDesc map[string]string) ([]string, error) {
	keyList := []string{"yourRsaPublicKey", "yourRsaPrivateKey"}
	return keyList, nil
}

// Decrypt objects encrypted with different master keys.
func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance.
	// Set yourEndpoint to the endpoint of the bucket. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. For more information about other regions, see the actual documentation.
	// Set yourRegion to the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to cn-hangzhou. For more information about other regions, see the actual documentation.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Set the signature version.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

	// Create a description for the master key. The description cannot be modified after it is created. The master key description and the master key have a one-to-one mapping.
	// If all objects use the same master key, the master key description can be empty. However, you cannot change the master key later.
	// If the master key description is empty, you cannot determine which master key is used for decryption.
	// We strongly recommend that you configure a master key description (a JSON string) for each master key and have the client save the mapping between the master key and its description. The server does not save this mapping.

	// A map converted from the master key description (a JSON string).
	materialDesc := map[string]string{
		"desc": "your master encrypt key material describe information",
	}

	// Create a master key object based on the master key description.
	// Set yourRsaPublicKey to the public key information of the master key that you manage, and set yourRsaPrivateKey to the private key information of the master key that you manage.
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		log.Fatalf("Error creating master RSA cipher: %v", err)
	}

	// Create an encryption interface based on the master key object and use AES-CTR for encryption.
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// To decrypt objects encrypted with different master keys, provide this interface.
	var mockRsaManager MockRsaManager
	var options []osscrypto.CryptoBucketOption
	options = append(options, osscrypto.SetMasterCipherManager(&mockRsaManager))

	// Obtain a created bucket for client-side encryption.
	// A client-side encrypted bucket is used in a similar way to a regular bucket.
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider, options...)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

	// Automatic encryption during PutObject.
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		log.Fatalf("Error putting object: %v", err)
	}

	// Automatic decryption during GetObject.
	body, err := cryptoBucket.GetObject("otherObjectNameEncryptedWithOtherRsa")
	if err != nil {
		log.Fatalf("Error getting object: %v", err)
	}
	defer body.Close()

	data, err := io.ReadAll(body)
	if err != nil {
		log.Fatalf("Error reading object data: %v", err)
	}
	log.Printf("Data: %s", string(data))
}

Upload and download an object using a KMS master key

The following sample code shows how to upload and download an object using a KMS master key:

package main

import (
	"bytes"
	"io"
	"log"

	"github.com/aliyun/alibaba-cloud-sdk-go/services/kms"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	crypto "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error creating credentials provider: %v", err)
	}

	// Create an OSSClient instance.
	// Set yourEndpoint to the endpoint of the bucket. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. For more information about other regions, see the actual documentation.
	// Set yourRegion to the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to cn-hangzhou. For more information about other regions, see the actual documentation.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Set the signature version.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error creating OSS client: %v", err)
	}

	// Create a KMS Client instance.
	kmsClient, err := kms.NewClientWithAccessKey("yourKmsRegion", "yourKmsAccessKeyId", "yourKmsAccessKeySecret")
	if err != nil {
		log.Fatalf("Error creating KMS client: %v", err)
	}

	// Create a description for the master key. The description cannot be modified after it is created. The master key description and the master key have a one-to-one mapping.
	// If all objects use the same master key, the master key description can be empty. However, you cannot change the master key later.
	// If the master key description is empty, you cannot determine which master key is used for decryption.
	// We strongly recommend that you configure a master key description (a JSON string) for each master key and have the client save the mapping between the master key and its description. The server does not save this mapping.

	// A map converted from the master key description (a JSON string).
	materialDesc := map[string]string{
		"desc": "your kms encrypt key material describe information",
	}

	// Create a master key object based on the master key description.
	// Set yourKmsId to the ID of the KMS user master key, which is the CMK ID.
	masterkmsCipher, err := crypto.CreateMasterAliKms(materialDesc, "yourKmsId", kmsClient)
	if err != nil {
		log.Fatalf("Error creating master KMS cipher: %v", err)
	}

	// Create an encryption interface based on the master key object and use AES-CTR for encryption.
	contentProvider := crypto.CreateAesCtrCipher(masterkmsCipher)

	// Obtain a created bucket for client-side encryption.
	// A client-side encrypted bucket is used in a similar way to a regular bucket.
	cryptoBucket, err := crypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		log.Fatalf("Error getting crypto bucket: %v", err)
	}

	// Automatic encryption during PutObject.
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		log.Fatalf("Error putting object: %v", err)
	}

	// Automatic decryption during GetObject.
	body, err := cryptoBucket.GetObject("yourObjectName")
	if err != nil {
		log.Fatalf("Error getting object: %v", err)
	}
	defer body.Close()

	data, err := io.ReadAll(body)
	if err != nil {
		log.Fatalf("Error reading object data: %v", err)
	}
	log.Printf("Data: %s", string(data))
}

References

For more information about the API operation to configure server-side encryption, see SetBucketEncryption.
For more information about the API operation to query server-side encryption configurations, see GetBucketEncryption.