Secure ACK Disk Volumes with KMS Encryption - Container Service for Kubernetes

Sensitive data in Kubernetes clusters is exposed to multiple attack surfaces: storage at rest, in-transit transmission, and in-memory processing. This topic describes how to use ACK with Alibaba Cloud Key Management Service (KMS) to encrypt disk volumes and manage Kubernetes Secrets securely, and provides best practices for hardening your cluster against key breaches.

Data encryption

Disk encryption recommendations

Enable the following practices to protect data stored on disk:

Enable disk encryption: Use encryption at rest as a security baseline. For more information, see Encrypt disk volumes.
Periodically rotate CMKs: Rotate customer master keys (CMKs) and configure key versioning to limit the blast radius of a compromised key. For more information, see Automatic key rotation.

Encrypt disk volumes with KMS

ACK supports scenarios with high security or compliance requirements. Using KMS-managed storage encryption protects data stored in ECS instances without requiring you to build or maintain an underlying key management system. For more information, see Encrypt disk volumes.

The following steps show how to use a KMS key to encrypt a disk volume at creation time.

Create a StorageClass.

Create a file named sc-kms.yaml with the following content:


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-disk
provisioner: diskplugin.csi.alibabacloud.com
parameters:
    fsType: ext4
    type: cloud_ssd
    encrypted: "true"
    kmsKeyId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
reclaimPolicy: Delete

Run the following command to create the StorageClass:
```
kubectl create -f sc-kms.yaml
```

Create a persistent volume claim (PVC).

Create a file named sc-pvc.yaml with the following content:


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: csi-disk

ReadWriteOnce

Run the following command to create the PVC:
```
kubectl create -f sc-pvc.yaml
```

Key management

Kubernetes Secrets let application developers store sensitive information—such as database passwords, application certificates, and tokens—as objects that pods can consume. Secrets have the following characteristics:

Secrets are scoped to namespaces. Use Secrets together with Kubernetes role-based access control (RBAC) to isolate reads and writes by namespace.
Mount Secrets as files or environment variables to containers in pods.
Sensitive information in Secrets is stored in the temporary file system (tmpfs) of nodes.
The API server saves Secrets as Base64-encoded plaintext in the etcd of the Kubernetes cluster.
A Secret is less than 1 MB in size.

Because Secrets do not inherently guarantee security, use cloud provider key management solutions and follow the security best practices described in this topic.

Security solutions

Based on the shared responsibility model for cloud security, cloud providers are responsible for securing configurations and sensitive information on the control plane and providing key management solutions. The following solutions are available.

Use KMS

Alibaba Cloud KMS provides professional key lifecycle management and data encryption and decryption services, including automatic key rotation, which reduces the risk of sensitive data breaches and helps enterprises meet compliance requirements.

To avoid hard-coding credentials in application code, use KMS to read and write keys throughout the pipeline of application development, testing, and building. For more information, see What is KMS?

Use the Secret Store CSI driver

Most applications retrieve keys from a specified file system path or environment variable. The secrets-store-csi-driver, developed by the Kubernetes community based on the CSI standard, mounts keys stored in an external key management service to pods using volumes, so applications access keys through the file system without creating a Kubernetes Secret or storing plaintext in etcd.

Capabilities:

Mounts keys stored in an external key management service to pods as volumes
Avoids storing plaintext Secrets in etcd and managing large numbers of Secrets in large clusters
Synchronizes key changes from KMS to containers in real time
Eliminates additional costs from calling the key management service in application code

Deploy the Alibaba Cloud secrets-store-csi-driver-provider-alibabacloud in your cluster to synchronize keys stored in KMS as files or Kubernetes Secrets to containers.

To resolve The Last Key issue (the CSI driver requires permissions to access keys in KMS), use RAM Roles for Service Accounts (RRSA). RRSA grants the service account of the Secret Store CSI driver access to KMS keys instead of granting permissions directly to pods.

Enable etcd encryption

etcd is the default storage system used by Kubernetes clusters. Secrets are stored as Base64-encoded plaintext in etcd by default, which poses security risks. In managed clusters, etcd is maintained by cloud service providers. Enable etcd encryption to automatically encrypt keys when syncing them from KMS to Secrets and decrypt keys when retrieving them. Combine etcd encryption with automatic key rotation to further enhance data security.

Use confidential containers

In scenarios that require enhanced data security—such as financial payment, privacy authentication, and data computing related to intellectual property protection—use ACK clusters for trusted execution environment (TEE)-based confidential computing. ACK for TEE-based confidential computing provides a trusted, encrypted execution environment based on the underlying hardware encryption technology. It protects the integrity and confidentiality of sensitive information throughout its lifecycle. Keys used by application systems can be stored in enclaves—isolated environments similar to a hardware security module (HSM) in KMS—to reduce key transmission.

Security best practices

Based on the key management solutions provided by cloud providers, developers and operations and maintenance (O&M) engineers must also secure keys on the business side.

Apply RBAC

Follow the least privilege principle when assigning cluster access. Avoid issuing credentials with read and write permissions on global Secrets. Revoke cluster credentials that may be compromised.

Harden pod security

Configure security contexts to minimize pod capabilities based on business requirements. Avoid privileged configurations, shared host networks, or shared file systems. Use security policies to reject privilege assignments that violate your security policies during deployment. Use network policies to control east-west traffic between pods and reduce the risk of lateral movement attacks.

Harden node security

Use private networks and add access control list (ACL) rules to security groups to control inbound and outbound traffic. Define security compliance standards based on Multi-Level Protection Scheme (MLPS) or Alibaba Cloud Linux Security Hardening to strengthen identity verification, access control, security auditing, and intrusion prevention. For more information, see ACK security hardening based on MLPS.

Run baseline inspections regularly to identify high-risk configurations and patch vulnerabilities promptly.

Harden supply chain security

Prohibit hard-coding sensitive information in application templates, code repositories, and configuration files. Use key management services to centrally manage keys throughout the artifact supply chain. Implement automatic security inspection mechanisms to prevent data breaches across each component of the supply chain.

Audit and monitor key operations

Audit and log all key read, write, use, and lifecycle management operations to ensure that sensitive data operations are traceable. Set up runtime monitoring with alert rules for suspicious read and write operations on sensitive data and for compromised AccessKey pairs. Alerts and logs help O&M engineers quickly respond to key breaches, assess impacts, and minimize losses.

Use temporary tokens and rotate keys

Avoid static keys such as AccessKey pairs in application systems. Use temporary tokens instead. If an attacker compromises a temporary token, the token is only valid for a limited time, which minimizes the attack surface and gives you time to patch the vulnerability.

For keys stored in KMS, enable automatic key rotation or rotate keys periodically to enhance application security.

Use envelope encryption to protect the last key

Envelope encryption uses a Content Encryption Key (CEK) to encrypt your data locally, then encrypts the CEK with a Key Encryption Key (KEK) stored in KMS. This keeps your data encrypted and decrypted offline on the business side, avoids uploading keys to the cloud, and reduces data transmission costs in scenarios that involve large volumes of data. For more information, see Use envelope encryption to encrypt and decrypt local data.

The Last Key is a common issue in key management service-based encryption and decryption scenarios. To protect the KEK based on least privilege, use Alibaba Cloud Resource Access Management (RAM) to control access to the RAM credentials used to retrieve the KEK. Use automatically rotated temporary tokens to protect those credentials. Use an isolation mechanism such as RRSA to limit which applications can access the RAM credentials. For more information, see RRSA.