Automating Client-Side Encryption for Objects in OSS

In this post, I will show you how to automate the process of encrypting your data on your own machine, to ensure that your data never leaves your computer unencrypted.

By Léon Rodenburg

Moving parts or all of your data into the cloud can feel scary. You are entrusting someone else with your greatest ideas, personal information or family photos. For companies, it can be even more daunting to hand over a full transaction history, credit card details or a customer list. Without owning the machines and disks on which the data is stored and without access to the data centers where the hardware is placed, it is near impossible to make sure that data is never seen by other parties. Luckily, cloud providers help you by providing the necessary tools for encrypting your data. By using these 'encryption-as-a-service' tools, you can at least know for sure that no data will ever be stored in plaintext. If someone breaks into the data center and finds a way to reach the hard disks, the encrypted blobs are useless. Depending on your industry, there might also be regulatory requirements to enforce encryption of all data at rest. Using the cloud provider's solution to enforce that can help you be compliant.

In Alibaba Cloud, that 'encryption-as-a-service' is called the Key Management Service (KMS). It allows you to generate customer master keys that you can select when your data needs to be encrypted. The authorization mechanisms in Alibaba Cloud can be used to be very specific about who can use what key in which situation. And because KMS integrates tightly with other services like the Object Storage Service (OSS) and ApsaraDB for RDS, enabling encryption on your files or database is often as simple as checking a box.

Your Key or Not Your Key

One thing is sometimes overlooked though: the keys that are used in services like KMS are generated by your cloud provider, who must keep the original keys so that they can be used to encrypt or decrypt your data at a later point in time. Who is to say that they don't use those keys to read your data while you are not accessing it? There is nothing that can guarantee that the keys are not used without your consent. So, you just have to trust your cloud provider. Depending on your cyber risk appetite however, you cannot or might not want to fully trust your cloud provider.

So what if you really want to store your data, but do not want to use the hosted encryption services available in the cloud? You can use client-side encryption! With client-side encryption, you create your own keys, encrypt your data on your own machine and only then transfer it into the cloud. No plaintext data will ever be sent over the wire and the keys will remain completely private. By managing encryption yourself, you can also make sure that the plaintext values are never written on disk and only ever live in the computer's memory, removing it as soon as you are done with the plaintext value.

Client-side encryption does add complexity to your setup and is not compatible with all cloud services. So there is a trade-off to be made: do you choose to trust your cloud provider and have the convenience of managed encryption made available to you, or do you encrypt critical data yourself and thereby choose to have complete control? There is no right and wrong answer unfortunately. Depending on several factors only you can identify, you might choose to do one, the other or a combination of both.

Encrypting & Decrypting Manually with OpenSSL

Let's say you have some data that you want to store in OSS buckets. Buckets have native support for KMS encryption, but because of good reasons, you choose to encrypt the data locally. What is the easiest way to do this?

If you have a single file, you can use OpenSSL to easily encrypt a file with a passphrase. Make sure you have OpenSSL installed, open a Terminal and fill in the following commands:

openssl enc -in <file> -out <output file> -aes-256-cbc
# enter aes-256-cbc encryption password: <type password>
# Verifying - enter aes-256-cbc encryption password: <type password>

Replace <file> with the file you want to encrypt and <output file> with the path where you want to store the encrypted file. OpenSSL will ask you for a passphrase that you will have to type twice. Remember the passphrase, as you will need it later when decrypting the file. OpenSSL will encrypt the file with an AES cipher with 256 bit key in cipher block chaining (CBC) mode).

The output file is ready to be uploaded to OSS. If you have an OSS bucket called biggest-secrets and the Alibaba Cloud CLI installed, uploading is as simple as this:

aliyun oss cp <output file> oss://biggest-secrets/<output file>

Replace <output file> with the filename of the encrypted file. You should see a confirmation that it was uploaded. If your bucket has a different name also be sure to replace that.

Remove the local files (<file> and <output file>). The encrypted file is stored securely in OSS now, without anyone having access to the data (even not Alibaba Cloud itself).

Next, let's retrieve and decrypt the file. To download the file, use this:

aliyun oss cp oss://biggest-secrets/<output file> <new file>

Replace <output file> with the file name you chose when uploading it. Replace <new file> with the name where you want to store the encrypted file locally.

Now, let's decrypt the file:

openssl enc -d -in <encrypted file> -out <decrypted file> -aes-256-cbc

Replace <encrypted file> with the path to the encrypted file you just downloaded. Replace <decrypted file> with the path where you want to store the decrypted contents. OpenSSL will ask for the passphrase that you entered when encrypting the contents. Type that passphrase, press Enter and the decrypted contents will be placed where you asked.

Automating Encryption in Python

Imagine encrypting and uploading 1500 images using the process above. It is not hard to see that it will be very tedious and error-prone. Luckily, most of the things in the world can be automated, and encryption is one of them. Let's see how you can encrypt a file in Python using the cryptography library.

The cryptography library supports all kinds of ciphers and protocols, but the easiest to work with and one that is relatively well-documented is Fernet. Just like AES, Fernet is a symmetric encryption. It uses the same key to encrypt and decrypt messages. It is also not possible to tamper with a message without having the original key. With the added benefit of being easy to use, it seems like a good choice for encrypting our files on OSS.

First make sure you have the cryptography library installed:

pip install cryptography

The basic flow of encryption using Fernet is really simple and just takes a few lines of Python:

from cryptography.fernet import Fernet

# Read a file's contents
with open("file.txt", "rb") as f:
  contents = f.read()

# Generate a key, keep this safe!
key = Fernet.generate_key()
print(key)

# Encrypt the contents
fernet = Fernet(key)
encrypted = fernet.encrypt(contents)
print(encrypted)

This should print the Fernet key on the first line and the encrypted contents of the file on the second line. Make sure file.txt exists or change it to the path of a file you would like to encrypt.

When generating a Fernet key, you are not asked to come up with the passphrase yourself. Fernet will generate 32 random characters as the key and return it to you in a Base64 encoding. This is much safer than coming up with your own key as that will probably be too short and too easy to remember (and thus guess).

Decrypting the file is very similar:

key = b'...'
encrypted = b'...'

fernet = Fernet(key)
decrypted = fernet.decrypt(encrypted)
print(decrypted)

Replace the key and encrypted string with the output of the previous code snippet. This should print the file's contents back to you in unencrypted form.

Working with a CLI

Automating the encryption using Python already enables you to encrypt more files in one script, but it is still a lot of work to create tools that can encrypt and upload complete directories. Luckily, I have added support for client-side encryption to the unofficial Ali CLI that can automate some workflows when working with Alibaba Cloud. This is how it works:

First, we need to generate a keyfile:

ali crypto generate-key -k KEYFILE

Replace KEYFILE with the name of the file where you want to store the key. The Fernet key that is generated is put there for usage later. Keep this file safe!

To encrypt and upload a directory of files to your bucket, use this:

ali crypto oss cp LOCAL_PATH oss://BUCKET/PATH -k KEYFILE

Replace the paths and the name of the keyfile. If the local and remote paths are directories, all the files in the directory will be encrypted and uploaded (excluding subdirectories). You can also specify the --recursive option to traverse all subdirectories and upload them in one go.

Downloading and decrypting the files is just as easy:

ali crypto oss cp oss://BUCKET/PATH LOCAL_PATH -k KEYFILE

This will copy all the files in the bucket to your machine and decrypt them using the keyfile.

Wrapping Up

And that's it! By using automated ways of encrypting your data on your own machine, you can guarantee that your data never leaves your computer unencrypted. This will make it impossible for cloud providers or other third parties managing your data to see the contents of your files, although it does have the added overhead of key management. Whether or not that is an acceptable overhead depends on your cyber risk appetite and the trust you place in your hosting provider. If you choose to use client-side encryption, it is of the utmost important that you properly manage the keyfiles. All your files will be lost if you lose access to the encryption key, so be sure to put it somewhere durable.

If you are curious how the encryption is implemented in Ali CLI, you can check out the source code here:

https://github.com/leonrodenburg/ali-cli/tree/master/ali/commands/crypto

Community

Automating Client-Side Encryption for Objects in OSS

Your Key or Not Your Key

Encrypting & Decrypting Manually with OpenSSL

Automating Encryption in Python

Working with a CLI

Wrapping Up

Léon Rodenburg

You may also like

Comments

Léon Rodenburg

Related Products

Data Encryption Service

Key Management Service

Database Security Solutions

Quick Starts