How to encrypt data and manage keys - Container Service for Kubernetes

Encrypt disks, secure Kubernetes Secrets, and rotate keys to protect sensitive data in ACK clusters. Continuously monitor and update encryption algorithms and key management strategies to adapt to evolving threats.

Data encryption

Disk encryption recommendations

Protect data at rest:

Enable disk encryption: Encrypt disk volumes as a security baseline.
Periodically rotate CMKs: Configure automatic key rotation and key versioning to limit the blast radius of a compromised key.

Encrypt disk volumes with KMS

Key Management Service (KMS) storage encryption protects data in Elastic Compute Service (ECS) instances without requiring a separate key management system. See Encrypt disk volumes.

To encrypt a disk volume with a KMS key at creation time:

Create a StorageClass.

Create a file named sc-kms.yaml:


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-disk
provisioner: diskplugin.csi.alibabacloud.com
parameters:
    fsType: ext4
    type: cloud_ssd
    encrypted: "true"
    kmsKeyId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
reclaimPolicy: Delete

Create the StorageClass:
```
kubectl create -f sc-kms.yaml
```

Create a persistent volume claim (PVC).

Create a file named sc-pvc.yaml:


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: csi-disk

ReadWriteOnce

Create the PVC:
```
kubectl create -f sc-pvc.yaml
```

Key management

Kubernetes Secrets store sensitive data—such as passwords, certificates, and tokens—as objects consumed by pods:

Secrets are namespace-scoped. Use Kubernetes role-based access control (RBAC) to isolate Secret reads and writes by namespace.
Mount Secrets as files or environment variables to containers in pods.
Secret data resides in the temporary file system (tmpfs) on nodes.
The API server stores Secrets as Base64-encoded plaintext in etcd.
A Secret is limited to 1 MB.

Secrets do not guarantee security by default. Use cloud key management solutions and the following best practices to protect sensitive data.

Security solutions

Under the shared responsibility model, cloud providers secure the control plane and provide key management solutions.

Use KMS

KMS manages key lifecycles and provides data encryption, including automatic key rotation, to reduce breach risks and meet compliance requirements.

To avoid hard-coding credentials, use KMS to manage keys across development, testing, and build pipelines. See What is KMS?

Use the Secret Store CSI driver

Most applications retrieve keys from a file system path or environment variable. The secrets-store-csi-driver, a Kubernetes community project, mounts keys from an external key management service to pods as volumes, so applications access keys without creating Kubernetes Secrets or storing plaintext in etcd.

Capabilities:

Mounts keys from an external key management service to pods as volumes
Avoids plaintext Secrets in etcd and reduces Secret management overhead in large clusters
Syncs key changes from KMS to containers in real time
Eliminates key management service API calls from application code

Deploy the Alibaba Cloud secrets-store-csi-driver-provider-alibabacloud to sync KMS keys as files or Kubernetes Secrets to containers.

To resolve The Last Key issue (the CSI driver needs KMS access permissions), use RAM Roles for Service Accounts (RRSA). RRSA grants the CSI driver's service account access to KMS keys instead of granting permissions directly to pods.

Enable etcd encryption

etcd stores Kubernetes Secrets as Base64-encoded plaintext by default, posing security risks. In managed clusters, cloud providers maintain etcd. Based on cloud zero-trust principles and compliance requirements in many scenarios, enable etcd encryption to encrypt keys when syncing from KMS to Secrets and decrypt them on retrieval. Combine with automatic key rotation for stronger data security.

Use confidential containers

For scenarios requiring enhanced data security—such as financial payment, privacy authentication, and intellectual property data computing—use ACK clusters for trusted execution environment (TEE)-based confidential computing. TEE-based confidential computing provides a trusted, encrypted execution environment based on hardware encryption, protecting sensitive data integrity and confidentiality throughout its lifecycle. Application keys can be stored in enclaves—isolated environments similar to a hardware security module (HSM) in KMS—to reduce key transmission.

Security best practices

Beyond cloud provider solutions, developers and operations and maintenance (O&M) engineers must secure keys at the application layer.

Apply RBAC

Follow the least privilege principle when assigning cluster access. Avoid issuing credentials with read and write permissions on global Secrets. Revoke cluster credentials that may be compromised.

Harden pod security

Container escape is a common attack against Kubernetes clusters. After an attacker escapes to the host, they can read Secrets stored on the node and escalate to cluster-wide access. Configure security contexts to minimize pod capabilities. Configure security contexts to minimize pod capabilities. Avoid privileged configurations, shared host networks, or shared file systems. Use security policies to reject privilege-violating deployments. Use network policies to control east-west pod traffic and reduce lateral movement attack risk.

Harden node security

Use private networks and add access control list (ACL) rules to security groups to control inbound and outbound traffic. Define compliance standards based on Multi-Level Protection Scheme (MLPS) or Alibaba Cloud Linux Security Hardening for identity verification, access control, auditing, and intrusion prevention. See ACK security hardening based on MLPS.

Run baseline inspections regularly to identify high-risk configurations and patch vulnerabilities promptly.

Harden supply chain security

Prohibit hard-coding sensitive information in templates, repositories, and configuration files. Use key management services to centrally manage keys across the supply chain. Implement automatic security inspections to prevent data breaches at each supply chain stage.

Audit and monitor key operations

Audit and log all key operations—read, write, use, and lifecycle management—to ensure traceability. Set up runtime monitoring with alerts for suspicious operations on sensitive data and compromised AccessKey pairs. Alerts and logs help O&M engineers respond to breaches, assess impacts, and minimize losses.

Use temporary tokens and rotate keys

Avoid static keys such as AccessKey pairs. Use temporary tokens instead. Compromised temporary tokens are only valid for a limited time, minimizing the attack surface and giving you time to patch the vulnerability.

For KMS keys, enable automatic key rotation or rotate keys periodically to enhance security.

Use envelope encryption to protect the last key

Envelope encryption uses a Content Encryption Key (CEK) to encrypt data locally, then encrypts the CEK with a Key Encryption Key (KEK) in KMS. This keeps encryption and decryption offline, avoids uploading keys to the cloud, reduces data transmission costs for large data volumes, and addresses concerns about trusting cloud providers with plaintext data. See Use envelope encryption to encrypt and decrypt local data.

The Last Key is a common issue in KMS-based encryption scenarios. To protect the KEK with least privilege, use Alibaba Cloud Resource Access Management (RAM) to control access to credentials used to retrieve the KEK. Use automatically rotated temporary tokens to protect those credentials. Use an isolation mechanism such as RRSA to limit which applications can access RAM credentials.