All Products
Search
Document Center

Object Storage Service:Client-side encryption in Go

Last Updated:Mar 12, 2024

If client-side encryption is enabled, objects are locally encrypted before they are uploaded to Object Storage Service (OSS). Only the holder of the customer master key (CMK) can decrypt the objects. This way, data security is enhanced during transmission and storage.

Disclaimer

  • When you use client-side encryption, you must ensure the integrity and validity of the customer master key (CMK). If the CMK is incorrectly used or lost due to improper maintenance, you are responsible for all losses and consequences caused by decryption failures.

  • When you copy or migrate encrypted data, you must ensure the integrity and validity of the object metadata related to client-side encryption. If the encrypted metadata is incorrectly used or lost due to improper maintenance, you are responsible for all losses and consequences caused by decryption failures.

Scenarios

  • Highly sensitive data: You want to encrypt sensitive data, such as Personal Identifiable Information (PII), transaction records, and health data, before the data leaves your local environment. This ensures effective protection of your data even when the data is intercepted.

  • Regulatory compliance: Regulations, such as Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR), require encryption of data stored on third-party platforms. Client-side encryption meets these requirements because CMKs are managed by users and not passed over networks or to cloud service providers.

  • Strong permission control: Your enterprise or developer may want to have full control over the encryption process, including the selection of encryption algorithms and the management and rotation of keys. In this case, you can use client-side encryption to ensure that only authorized users can decrypt and access data.

  • Secure data migration across regions: Client-side encryption helps keep data encrypted throughout migration. This enhances data security during transmission over the Internet.

Usage notes

  • In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.

  • In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.

  • In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

Background information

In client-side encryption, a random data key is generated for each object to perform symmetric encryption on the object. The client uses a CMK to encrypt the random data key. The encrypted data key is uploaded as a part of the object metadata and stored in the OSS server. When an encrypted object is downloaded, the client uses the CMK to decrypt the random data key and then uses the data key to decrypt the object. To ensure data security, the CMK is used only on the client and is not transmitted over the network or stored in the server.

Important
  • Client-side encryption supports multipart upload for objects larger than 5 GB in size. When you use multipart upload to upload an object, you must specify the total size of the object and the size of each part. The size of each part except for the last part must be the same and a multiple of 16 bytes.

  • After you upload objects encrypted on the client, object metadata related to client-side encryption is protected, and cannot be modified by calling the CopyObject operation.

Encryption methods

You can use two types of CMKs for client-side encryption:

  • KMS-managed CMKs

    When you use a CMK managed by Key Management Service (KMS) for client-side encryption, you must send the CMK ID to OSS SDK for Python.

  • RSA-based CMKs managed by yourself

    When you use a CMK managed by yourself for client-side encryption, you must send the public key and the private key of your CMK to OSS SDK for Python as parameters.

You can use the preceding encryption methods to prevent data leaks and protect your data on the client. Even if your data is leaked, the data cannot be decrypted by other users.

Object metadata related to client-side encryption

Parameter

Description

Required

x-oss-meta-client-side-encryption-key

The encrypted data key. The encrypted data key is a string encrypted by using a CMK and encoded in Base64.

Yes

x-oss-meta-client-side-encryption-start

The initial value that is randomly generated for data encryption. The initial value is a string encrypted by using a CMK and encoded in Base64.

Yes

x-oss-meta-client-side-encryption-cek-alg

The algorithm that is used to encrypt data.

Yes

x-oss-meta-client-side-encryption-wrap-alg

The algorithm that is used to encrypt the data key.

Yes

x-oss-meta-client-side-encryption-matdesc

The description of the CMK in JSON format.

Warning

We recommend that you configure a description for each CMK and store the mapping relationship between the CMK and its description. A CMK without a matching description cannot be replaced.

No

x-oss-meta-client-side-encryption-unencrypted-content-length

The length of data before encryption. If content-length is not specified, this parameter is not generated.

No

x-oss-meta-client-side-encryption-unencrypted-content-md5

The MD5 hash of the data before encryption. If Content-MD5 is not specified, this parameter is not generated.

No

x-oss-meta-client-side-encryption-data-size

The total size of the object for which you want to perform multipart upload. If you want to encrypt the object on the client side, you must specify this parameter when you initialize a multipart upload task for the object.

Yes (for multipart upload)

x-oss-meta-client-side-encryption-part-size

The size of each part in a multipart upload task for the object. If you want to encrypt the object on the client side, you must specify this parameter when you initialize a multipart upload task for the object.

Note

The size of each part must be an integer multiple of 16 bytes.

Yes (for multipart upload)

Use RSA-based CMKs to encrypt objects in simple upload or decrypt objects in simple download

The following code provides an example on how to use an RSA-based CMK to encrypt objects in simple upload or decrypt objects in simple download:

package main

import (
	"bytes"
	"fmt"
	"io/ioutil"
	"os"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK. 
	// If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
	// If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
	// We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

	// Obtain the mapping relationship based on the CMK description that is a JSON string. 
	materialDesc := make(map[string]string)
	materialDesc["desc"] = "your master encrypt key material describe information"

	// Create a CMK object based on the description of the CMK. 
	// Set yourRsaPublicKey to the public key of the CMK that you manage, and set yourRsaPrivateKey to the private key of the CMK that you manage. 
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// Obtain an existing bucket to be used for client-side encryption. 
	// The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used. 
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Encryption is automatically performed when you call the PutObject operation. 
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Decryption is automatically performed when you call the GetObject operation. 
	body, err := cryptoBucket.GetObject("yourObjectName")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	defer body.Close()

	data, err := ioutil.ReadAll(body)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	fmt.Println("data:", string(data))
}

Use RSA-based CMKs to encrypt objects in multipart upload

The following code provides an example on how to use an RSA-based CMK to encrypt objects in multipart upload:

package main

import (
	"fmt"
	"os"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK. 
	// If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
	// If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
	// We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

	// Obtain the mapping relationship based on the CMK description that is a JSON string. 
	materialDesc := make(map[string]string)
	materialDesc["desc"] = "your master encrypt key material describe information"

	// Create a CMK object based on the description of the CMK. 
	// Set yourRsaPublicKey to the public key of the CMK that you manage, and set yourRsaPrivateKey to the private key of the CMK that you manage. 
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// Obtain an existing bucket to be used for client-side encryption. 
	// The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used. 
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	fileName := "yourLocalFilePath"
	fileInfo, err := os.Stat(fileName)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	fileSize := fileInfo.Size()

	// Encrypt the context. 
	var cryptoContext osscrypto.PartCryptoContext
	cryptoContext.DataSize = fileSize

	// Specify the number of parts that you want to encrypt. The actual number is based on the calculation result. 
	expectPartCount := int64(10)

	// Specify the size of a part encrypted in AES CTR mode, which is an integer multiple of 16 bytes. 
	cryptoContext.PartSize = (fileSize / expectPartCount / 16) * 16

	imur, err := cryptoBucket.InitiateMultipartUpload("yourObjectName", &cryptoContext)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	chunks, err := oss.SplitFileByPartSize(fileName, cryptoContext.PartSize)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	var partsUpload []oss.UploadPart
	for _, chunk := range chunks {
		part, err := cryptoBucket.UploadPartFromFile(imur, fileName, chunk.Offset, chunk.Size, (int)(chunk.Number), cryptoContext)
		if err != nil {
			fmt.Println("Error:", err)
			os.Exit(-1)
		}
		partsUpload = append(partsUpload, part)
	}

	// Complete
	_, err = cryptoBucket.CompleteMultipartUpload(imur, partsUpload)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
}

Decrypt objects that are encrypted by using different RSA-based CMKs

The following code provides an example on how to decrypt objects that are encrypted by using different RSA-based CMKs:

package main

import (
	"bytes"
	"fmt"
	"io/ioutil"
	"os"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

// Query the CMK based on the description of the CMK. To decrypt objects that are encrypted by using different CMKs, you must perform this operation. 
type MockRsaManager struct {
}

func (mg *MockRsaManager) GetMasterKey(matDesc map[string]string) ([]string, error) {
	keyList := []string{"yourRsaPublicKey", "yourRsaPrivatKey"}
	return keyList, nil
}

// Decrypt objects encrypted by using different CMKs. 
func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK. 
	// If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
	// If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
	// We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

	// Obtain the mapping relationship based on the CMK description that is a JSON string. 
	materialDesc := make(map[string]string)
	materialDesc["desc"] = "your master encrypt key material describe information"

	// Create a CMK object based on the description of the CMK. 
	// Set yourRsaPublicKey to the public key of the CMK that you manage, and set yourRsaPrivateKey to the private key of the CMK that you manage. 
	masterRsaCipher, err := osscrypto.CreateMasterRsa(materialDesc, "yourRsaPublicKey", "yourRsaPrivateKey")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
	contentProvider := osscrypto.CreateAesCtrCipher(masterRsaCipher)

	// To decrypt objects that are encrypted by using different CMKs, you must perform this operation. 
	var mockRsaManager MockRsaManager
	var options []osscrypto.CryptoBucketOption
	options = append(options, osscrypto.SetMasterCipherManager(&mockRsaManager))

	// Obtain an existing bucket to be used for client-side encryption. 
	// The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used. 
	cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider, options...)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Encryption is automatically performed when you call the PutObject operation. 
	err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Decryption is automatically performed when you call the GetObject operation. 
	body, err := cryptoBucket.GetObject("otherObjectNameEncryptedWithOtherRsa")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	defer body.Close()

	data, err := ioutil.ReadAll(body)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	fmt.Println("data:", string(data))
}

Use CMKs in KMS to encrypt objects in simple upload or decrypt objects in simple download

The following code provides an example on how to use a CMK in KMS to encrypt objects in simple upload or decrypt objects in simple download:

package main

import (
  "bytes"
  "fmt"
  "io/ioutil"
  "os"

  "github.com/aliyun/alibaba-cloud-sdk-go/services/kms"
  "github.com/aliyun/aliyun-oss-go-sdk/oss"
  "github.com/aliyun/aliyun-oss-go-sdk/oss/crypto"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

  // Create a KMSClient instance. 
  kmsClient, err := kms.NewClientWithAccessKey("yourKmsRegion", "yourKmsAccessKeyId", "yourKmsAccessKeySecret")
  if err != nil {
    fmt.Println("Error:", err)
    os.Exit(-1)
  }

  // Specify a description for the CMK. The description cannot be modified after being specified. You can specify only one description for a CMK. 
  // If all objects share a CMK, you can leave the description of the CMK empty. In this case, the CMK cannot be replaced. 
  // If the description of the CMK is left empty, the client cannot determine which CMK to use for decryption. 
  // We recommend that you configure a description in the form of a JSON string for each CMK and store the mapping relationship between the CMK and the description on the client. The server does not store the mapping relationship. 

    // Obtain the mapping relationship based on the CMK description that is a JSON string. 
  materialDesc := make(map[string]string)
  materialDesc["desc"] = "your kms encrypt key material describe information"

  // Create a CMK object based on the description of the CMK. 
  // Set yourKmsId to the ID of the CMK in KMS. 
  masterkmsCipher, err := osscrypto.CreateMasterAliKms(materialDesc, "yourKmsId", kmsClient)
  if err != nil {
    fmt.Println("Error:", err)
    os.Exit(-1)
  }

  // Create an encryption operation based on the CMK object and perform encryption in AES CTR mode. 
  contentProvider := osscrypto.CreateAesCtrCipher(masterkmsCipher)

  // Obtain an existing bucket to be used for client-side encryption. 
  // The bucket for which client-side encryption is configured is used in the same manner in which a common bucket is used. 
  cryptoBucket, err := osscrypto.GetCryptoBucket(client, "yourBucketName", contentProvider)
  if err != nil {
    fmt.Println("Error:", err)
    os.Exit(-1)
  }

  // Encryption is automatically performed when you call the PutObject operation. 
  err = cryptoBucket.PutObject("yourObjectName", bytes.NewReader([]byte("yourObjectValueByteArrary")))
  if err != nil {
    fmt.Println("Error:", err)
    os.Exit(-1)
  }

  // Decryption is automatically performed when you call the GetObject operation. 
  body, err := cryptoBucket.GetObject("yourObjectName")
  if err != nil {
    fmt.Println("Error:", err)
    os.Exit(-1)
  }
  defer body.Close()

  data, err := ioutil.ReadAll(body)
  if err != nil {
    fmt.Println("Error:", err)
    os.Exit(-1)
  }
  fmt.Println("data:", string(data))
}