Use Vault as a key management service in ACK clusters - Container Service for Kubernetes

Vault is an identity-based key management and encryption system. Vault can secure, store, and control access to sensitive data such as tokens, passwords, certificates, and API keys. It resolves the issue of sensitive information hardcoding in applications. This topic describes how to deploy and use Vault in a Container Service for Kubernetes (ACK) cluster.

Prerequisites

An ACK cluster is created and the version of the cluster is 1.22 or later. For more information, see Create an ACK managed cluster and Manually update ACK clusters.
Helm is installed and the version of Helm is 3.6 or later. For more information, see Helm Release.

Install and initialize Vault

Step 1: Install Vault

Use one of the following methods to obtain the Vault installation package. In this topic, the Helm chart version is vault-0.24.1 and the Vault version is 1.3.1.
- Log on to Git repository vault-helm to obtain the package.
- Remotely download the package from the repository. To do this, run the following command to add and update the repository:
```
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
```
Run the following command to assign an alias to simplify the configuration.
In the following sample code, the KubeConfig of the ACK cluster is stored in $HOME/Downloads/kubeconfig. Replace it with the actual path of the KubeConfig.
```
# Helm client 
alias h="helm --kubeconfig $HOME/Downloads/kubeconfig"
# kubectl client 
alias k="kubectl --kubeconfig $HOME/Downloads/kubeconfig"
```
Install Vault.
We recommend that you install Vault of the Raft version in a production environment. Do not install the standalone version in a production environment.
1. Run the following command to create a namespace named vault:
```
k create ns vault
```
  Install Vault in the vault namespace. In subsequent operations, set the Kubernetes namespace to vault.
2. Run the following command to install Vault in the vault namespace.
  A StorageClass supported by ACK is used in the following command. You can run the k get sc command to query the StorageClass. The volume must be 20 GiB or larger. After you deploy Vault, three pay-as-you-go enhanced SSDs (ESSDs) are generated and mounted as persistent volumes (PVs) to the Vault pods. For information about disk billing, see Billing.
```
h install vault -nvault hashicorp/vault \
    --set='server.ha.enabled=true' \
    --set='server.ha.raft.enabled=true' \
    --set='server.dataStorage.size=20Gi' \
    --set='server.dataStorage.storageClass=alicloud-disk-essd'
```
3. Run the following command to query the status of the Vault pods:
```
k get po -n vault
```
  Expected output:
```
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          45s
vault-1                                 0/1     Running   0          45s
vault-2                                 0/1     Running   0          44s
vault-agent-injector-59fdd7cdf8-prwv7   1/1     Running   0          45s
```

Step 2: Initialize and unseal Vault

Run the following command to query the initial status of Vault after Vault is launched:

k exec -nvault vault-0 -- vault status

Expected output:

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            1.13.1
Build Date         2023-03-23T12:51:35Z
Storage Type       raft
HA Enabled         true
command terminated with exit code 2

If Initialized displays false and Sealed displays true, Vault is not initialized or unsealed. You need to perform the following steps to initialize and unseal Vault.

Run the following command to initialize Vault.
Generate a key from the Vault binary in the container.
```
k exec vault-0 -nvault -- vault operator init -key-shares=5 -key-threshold=3 -format=json > cluster-keys.json
```
During the initialization process, the system generates 5 shares and sets the unseal_threshold to 3.
In the production environment, the pods send POST API requests whose path is vault-0.vault.vault.svc:8200/sys/init to initialize Vault.
In this example, the DNS resolution convention for StatefulSets is used, which is {$podName}.{$stsName}.{$Namespace}.svc. For more information about the API used to initialize Vault, see Vault Start Initialization.
View the content of the cluster-keys.json file, which is similar to the data returned by an API
```
{
  "unseal_keys_b64": [
    "Zu6EdLIFn+2****",
    "yvBur6WEphY****",
    "HR4hNkQN++h****",
    "85jAjs8xAj4****",
    "DgYQhjo6l14****"
  ],
  "unseal_keys_hex": [
    "66ee8474b****",
    "caf06eafa****",
    "1d1e21364****",
    "f398c08ec****",
    "0e0610863****"
  ],
  "unseal_shares": 5,
  "unseal_threshold": 3,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 0,
  "recovery_keys_threshold": 0,
  "root_token": "hvs.5aiXKN****"
}
```
Export unseal_keys_b64 from the preceding cluster-keys.json file and perform the following steps to unseal Vault.
Run the following command to unseal Vault. For more information about the API used to unseal Vault, see Vault Unseal.
In the preceding step, unseal_threshold is set to 3. Therefore, three unseal keys are used to perform the unseal operation three times.
```
k exec -nvault vault-0 -- vault operator unseal Zu6EdLIFn+2****
```

Run the following command to query the status of vault-0:

k exec -it vault-0 -n vault -- vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.13.1
Build Date              2023-03-23T12:51:35Z
Storage Type            raft
Cluster Name            vault-cluster-504959a1
Cluster ID              d99594a5-75de-53fa-59dd-19ed024b****
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2023-05-06T10:30:38.237415781Z
Raft Committed Index    36
Raft Applied Index      36

k get po -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          46m
vault-1                                 0/1     Running   0          46m
vault-2                                 0/1     Running   0          46m
vault-agent-injector-59fdd7cdf8-prwv7   1/1     Running   0          46m

The output indicates that vault-0 is initialized.

(Optional) Use the root token to log on to and check the Raft node.

Run the following command to log on to the vault-0 node.
In this topic, the value of root_token is hvs.5aiXKN****. You can obtain the root token from the cluster-keys.json file generated in Step 3.
```
k exec vault-0 -n vault -- vault login hvs.5aiXKN****
```

Run the following command to query the Raft node:

k exec -nvault vault-0 -- vault operator raft list-peers

Expected output:

Node                                    Address                        State     Voter
----                                    -------                        -----     -----
10285056-839a-f306-a301-5024934a794f    vault-0.vault-internal:8201    leader    true

Step 3: Add follower nodes

Run the following command to add Vault nodes. For more information about the API used to add Vault nodes, see Raft.

k exec -nvault vault-1  -- vault operator raft join http://vault-0.vault-internal:8200
Key       Value
---       -----
Joined    true
k exec -nvault vault-2  -- vault operator raft join http://vault-0.vault-internal:8200
Key       Value
---       -----
Joined    true

Run the following commands to unseal and add Vault nodes.

Use the three unseal keys to unseal each node. This means that you need to perform the unseal operation six times.

k exec -nvault vault-1 -- vault operator unseal Zu6EdLIF****
k exec -nvault vault-2 -- vault operator unseal Zu6EdLIF****
...
k exec -nvault vault-1 -- vault operator unseal DgYQhjo6****
k exec -nvault vault-2 -- vault operator unseal DgYQhjo6****

Run the following command to check whether the Vault nodes are added:

k exec -n vault vault-0 -- vault operator raft list-peers

Expected output:

Node                                    Address                        State       Voter
----                                    -------                        -----       -----
10285056-839a-f306-a301-5024934a794f    vault-0.vault-internal:8201    leader      true
71ffd98c-d6d4-a7b3-994c-9ce87f464486    vault-1.vault-internal:8201    follower    true
1e9f37dc-b55b-fc46-8ca8-595428ad1d81    vault-2.vault-internal:8201    follower    true

k get po -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          66m
vault-1                                 1/1     Running   0          66m
vault-2                                 1/1     Running   0          66m
vault-agent-injector-59fdd7cdf8-prwv7   1/1     Running   0          66m

The output indicates that the vault-1 and vault-2 nodes are added.

Examples

Example 1: Use Vault to manage ServiceAccount tokens in ACK clusters

You can use Vault to obtain the rolebinding and clusterrolebinding tokens. After you configure Vault to obtain tokens, no Secrets are generated during role binding in the ACK cluster. This method uses Vault to obtain the bearer tokens of the API server to prevent attackers from obtaining ServiceAccount credentials from the ACK cluster.

Create a file named ClusterRole.yaml and a file named ClusterRoleBinding.yaml, and add the following content to the files.

View the content of the ClusterRole.yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-minimal-secrets-abilities
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]

View the content of the ClusterRoleBinding.yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-token-creator-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-minimal-secrets-abilities
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault

Run the following command to bind the ClusterRole to the ServiceAccount of Vault so that Vault can create a token for the ServiceAccount.
```
k apply -f ClusterRole.yaml
k apply -f ClusterRoleBinding.yaml
```
Run the following command to enable Kubernetes Secrets for Vault:
```
k exec -nvault vault-0 -- vault secrets enable kubernetes
```

Verify the configuration.

Run the following command to create a namespace named test:
```
k create ns test
```

Create a file named test.yaml and add the following content to the file:

View the content of the test.yaml file

apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-service-account-with-generated-token
  namespace: test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: test-role-list-pods
  namespace: test
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: test-role-abilities
  namespace: test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: test-role-list-pods
subjects:
- kind: ServiceAccount
  name: test-service-account-with-generated-token
  namespace: test

Run the following command to deploy the test.yaml file to generate the ServiceAccount RoleBinding for testing purpose:
```
k apply -f test.yaml
```

Run the following command to query the token: For more information about the API, see Kubernetes API.

k exec -nvault vault-0 -- vault write -f kubernetes/config
k exec -nvault vault-0 -- vault write kubernetes/roles/my-role allowed_kubernetes_namespaces="*" service_account_name="test-service-account-with-generated-token" token_default_ttl="10m"
k exec -nvault vault-0 -- vault write kubernetes/creds/my-role kubernetes_namespace=test

The JSON web token generated by the last command can be used to access the API server.

The service_account_token field in the following code displays the JSON web token:

Key                          Value
---                          -----
lease_id                     kubernetes/creds/my-role/XPDLbuXJ0Bt4fF****
lease_duration               10m
lease_renewable              false
service_account_name         test-service-account-with-generated-token
service_account_namespace    test
service_account_token        eyJhbGciOiJSUzI1NiIsImtp****

Run the following command to access the API server:

curl -sk https://XX.XX.XX.XX:6443/api/v1/namespaces/test/pods --header "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtp****"
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "2861371"
  },
  "items": []
}

The validity period of the preceding token is 10 minutes. If the token expires, you need to recall the write API to obtain another token.

k exec -nvault vault-0 -- vault write kubernetes/creds/my-role kubernetes_namespace=test

Example 2: Dynamically obtain RAM credentials in a pod

You can use Vault to store and access the AccessKey IDs and AccessKey secrets of Resource Access Management (RAM) users. Applications interact with Vault to dynamically obtain the required credentials. For more information about the corresponding API, see AliCloud Secrets Engine.

Run the following command to enable Alicloud Secrets for Vault:
```
k exec -nvault vault-0 -- vault secrets enable  alicloud
```
Log on to the RAM console with your Alibaba Cloud account.
Create a custom policy named ExampleRAMPolicyforVault and add the following content to the policy. For more information, see Create a custom policy.
This policy allows RAM users to assume any types of roles to create and delete credentials or policies, attach policies to other RAM users, detach policies, create and delete RAM users, and access resources.
View the content of the custom policy named ExampleRAMPolicyforVault
```
{
  "Statement": [
    {
      "Action": [
        "ram:CreateAccessKey",
        "ram:DeleteAccessKey",
        "ram:CreatePolicy",
        "ram:DeletePolicy",
        "ram:AttachPolicyToUser",
        "ram:DetachPolicyFromUser",
        "ram:CreateUser",
        "ram:DeleteUser",
        "sts:AssumeRole"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ],
  "Version": "1"
}
```
Create a RAM user named hashicorp-vault. For more information, see Create a RAM user.
Attach the ExampleRAMPolicyforVault policy to the hashicorp-vault RAM user. For more information, see Grant permissions to a RAM user.
Create an AccessKey pair for the RAM user. For more information, see Create an AccessKey.
Record the AccessKey ID and AccessKey secret. In this example, the AccessKey ID is ak1 and the AccessKey secret is sk1.
Run the following command to write the AccessKey pair to Vault.
```
k exec -nvault vault-0 -- vault write alicloud/config access_key=ak1 secret_key=sk
Success! Data written to: alicloud/config
```
The AccessKey pair is saved to the /vault/data/vault.db file on each Vault node. The file is persisted to volumes. Therefore, the file is not lost after the nodes are restarted.

Write remote and inline policies to Vault.

Run the following command to write remote policies to Vault.

Remote policies refer to existing policies in RAM. In this example, a custom policy named ExampleRAMPolicyforVault, a system policy named AliyunOSSReadOnlyAccess, and a system policy named AliyunRDSReadOnlyAccess are written to Vault.

k exec -nvault vault-0 -- vault write alicloud/role/policy-based \
    remote_policies='name:ExampleRAMPolicyforVault,type:Custom' \
    remote_policies='name:AliyunOSSReadOnlyAccess,type:System' \
    remote_policies='name:AliyunRDSReadOnlyAccess,type:System'
Success! Data written to: alicloud/role/policy-based

Run the following command to write inline policies to Vault.

Inline policies refer to policies that are written in API requests. In this example, the custom policy named ExampleRAMPolicyforVault is written to Vault.

k exec -nvault vault-0 -- vault write alicloud/role/policy-based \
    inline_policies=-<<EOF
[
{
  "Statement": [
    {
      "Action": [
        "ram:CreateAccessKey",
        "ram:DeleteAccessKey",
        "ram:CreatePolicy",
        "ram:DeletePolicy",
        "ram:AttachPolicyToUser",
        "ram:DetachPolicyFromUser",
        "ram:CreateUser",
        "ram:DeleteUser",
        "sts:AssumeRole"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ],
  "Version": "1"
}
]
EOF

Create a RAM role named vaultTestRole and attach the ExampleRAMPolicyforVault policy to the RAM role. For more information, see Create a RAM role for a trusted Alibaba Cloud account and Grant permissions to a RAM user.
After the policy is attached to the RAM role, an Alibaba Cloud Resource Name (ARN) is generated. The string 15261**** indicates the ID of the RAM user.
```
vaultTestRole@role.15261****.onaliyunservice.com
```

Run the following command to write the ARN to Vault. This allows Vault to assume the vaultTestRole role.

k exec -nvault vault-0 -- vault write alicloud/role/role-based \
      role_arn='acs:ram::15261****:role/vaultTestRole'

Verify the configuration

Run the following command to query the credentials generated based on policies:

k exec -nvault vault-0 -- vault read alicloud/creds/policy-based
Key                Value
---                -----
lease_id           alicloud/creds/policy-based/TG1isE6uga94sRv60NK7****
lease_duration     768h
lease_renewable    true
access_key         ak1
secret_key         sk1

Run the following command to query the credentials (Security Token Service token) generated based on roles:

k exec -nvault vault-0 -- vault read alicloud/creds/role-based
Key                Value
---                -----
lease_id           alicloud/creds/role-based/uJxVwNSnqzcni75kkf****
lease_duration     59m59s
lease_renewable    false
access_key         STS.NUM2e1BrC****
expiration         2023-05-09T04:16:17Z
secret_key         3VmmRy****
security_token     CAISiwJ1q****

Permissions

The following table describes the permissions provided by different policies that are attached to the same role.

Policy	Permission
inline_policies	"ram:CreateAccessKey" "ram:DeleteAccessKey" "ram:AttachPolicyToUser" "ram:DetachPolicyFromUser" "ram:CreateUser" "ram:DeleteUser"
remote_policies	All permissions provided by inline policies "ram:CreatePolicy" "ram:DeletePolicy"
role_arn	"sts:AssumeRole"

Example 3: Obtain RAM credentials in a pod to access the Vault APIs

You can access Vault from a pod by including the identity authentication information in the code of the corresponding application. This method can be used in Alibaba Cloud identity authentication, identity query, and role query. The application that carries the identity authentication information call the Vault APIs (Auth, Alicloud, and Login) to obtain the role of the corresponding user and the token that is used to access Vault.

The obtained token can be used to perform operations on the corresponding role. For example, the token can be used to query detailed information about the role, query a list of roles, create roles, and delete roles. For more information, see AliCloud Auth Method (API).

Run the following command to enable Alicloud authentication for Vault. For more information about the API, see Enable Auth Method.
```
k exec -nvault vault-0 -- vault auth enable alicloud
```

Run the following command to write the ARN to Vault:

k exec -nvault vault-0 -- vault write auth/alicloud/role/vaultTestRole arn='acs:ram::15261****:role/vaulttestrole'

Obtain the URL and header that are used to call the GetCallerIdentity API. For more information, see vault-plugin.
The application needs to call the GetCallerIdentity API and then obtains the URL and header that are used to call the API. For more information about the GetCallerIdentity API, see GetCallerIdentity OpenAPI. On the OpenAPI Portal page, enter GetCallerIdentity into the left-side search box, select the region in which the application is deployed in the middle part of the page, and then click the SDK Sample tab on the right side.
Call the Login API. IDENTITY_REQUEST_URL_BASE_64 is a Base64-encoded URL. IDENTITY_REQUEST_HEADERS_BASE_64 is a Base64-encoded header. For more information, see Login OpenAPI.
After the application calls the Login API, the client_token field is returned. Then, the application can call the Auth and Alicloud APIs of Vault to assume the corresponding roles.

FAQ

Vault is a third-party maintained open-source project. For issues with products or components that are not officially supported by Alibaba Cloud or ACK, see Vault Community.

How do I fix Vault cluster anomalies?

If the Vault pods restart, the pods enter the 0/1 Running state. You can perform the following operations to fix Vault cluster anomalies.

Run the following command to query the status of the Vault pods:

k exec -nvault vault-0 -- vault status

Expected output:

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.13.1
Build Date         2023-03-23T12:51:35Z
Storage Type       raft
HA Enabled         true
command terminated with exit code 2

The output indicates that the pods are in the sealed state. You need to unseal the corresponding Vault node again.

Run the following command to unseal the Vault node.
Use the three unseal keys to unseal the node. This means that you need to perform the unseal operation three times on the node.
```
k exec -nvault vault-0 -- vault operator unseal Zu6EdL****
```

Run the following command to use the root token to log on to a node and view the Raft list:

k exec -nvault vault-0 -- vault operator raft list-peers

Expected output:

Node                                    Address                        State       Voter
----                                    -------                        -----       -----
10285056-839a-f306-a301-5024934a794f    vault-0.vault-internal:8201    follower    true
71ffd98c-d6d4-a7b3-994c-9ce87f464486    vault-1.vault-internal:8201    leader      true
1e9f37dc-b55b-fc46-8ca8-595428ad1d81    vault-2.vault-internal:8201    follower    true

The output indicates that the vault-0 node is in the follower state, which is normal.

How do I use a Service in an ACK cluster to call Vault?

Multiple Services are generated after Vault is installed. Run the following command to query the Services:

k get svc -n vault

Expected output:

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
vault                      ClusterIP   172.16.193.219   <none>        8200/TCP,8201/TCP   47h
vault-active               ClusterIP   172.16.177.54    <none>        8200/TCP,8201/TCP   47h
vault-internal             ClusterIP   None             <none>        8200/TCP,8201/TCP   47h
vault-standby              ClusterIP   172.16.29.54     <none>        8200/TCP,8201/TCP   47h

vault and vault-internal are LoadBalancer Services and vault-internal is a headless Service.
vault-active is a leader node elected by Raft.
vault-standby is a follower node of Raft.