Build a Redis cluster on an elastic container instance - Elastic Container Instance

Containerized Redis clusters are easy to deploy and scale and store resources in isolated space. This improves your efficiency in deploying, managing, scaling up, and maintaining Redis clusters and reduces O&M costs of Redis clusters. This topic describes how to build a Redis cluster on an elastic container instance in a Container Service for Kunbernetes (ACK) Serverless cluster.

Background information

Redis instances can run in cluster mode. In this mode, Redis divides the storage space into 16,384 hash slots. Each master node in the Redis cluster is responsible for a database shard, which consists of a specific number of hash slots. When you write a data entry, Redis calculates hash slots for the data entry and then writes the data to the node that is responsible for the hash slots. You can add one or more slave nodes to each master node. When a master node is unavailable, one of the slave nodes automatically takes over the master node to provide services.

Redis provides high throughput and availability in cluster mode. However, Redis still cannot prevent data loss in this mode. For more information, see Official Redis documentation.

Prerequisites

An ACK Serverless cluster that meets the following conditions is created:

Internet access is enabled for the cluster, and the cluster can pull public images.
A storage plug-in is deployed in the cluster.
We recommend that you deploy the Container Storage Interface (CSI) plug-in. Make sure that the Alibaba Cloud CSI-Provisioner component is deployed in the CSI plug-in.
CoreDNS is deployed in the cluster.

Build a Redis cluster

In this topic, the official Redis image v6.0.8 is used as an example to create a Redis cluster that consists of six nodes, including three master nodes and three slave nodes. Each node has a status and an identifier. You can use a Statefulset to create pods. You must attach a disk to each node to persistently store node data.

Note

We recommend that you use Redis v5.0 or later. If you use a Redis version earlier than 5.0, you may need to use a command that is different from the command provided in this topic to initialize the cluster.

Create a ConfigMap to store and manage the configurations of the Redis cluster.
```
kubectl create -f redis-config.yaml
```
Sample content of the redis-config.yaml file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
data:
  redis.conf: |
    bind 0.0.0.0
    port 6379
    cluster-announce-bus-port 16379
    cluster-enabled yes
    appendonly yes
    cluster-node-timeout 5000
    dir /data
    cluster-config-file /data/nodes.conf
    requirepass pass123
    masterauth pass123
```
- The dir field specifies the persistent storage directory of node data in the Redis cluster. The /data directory of pods must be a persistent storage directory.
- The cluster-config-file field specifies the node information of the Redis cluster. The node information is automatically generated and modified by the Redis node. The value of the field is also a persistent storage directory. After a node fails, other nodes in the cluster can take over the failed node to provide services.

Create a headless Service.

kubectl create -f redis-service.yaml

Sample content of the redis-service.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-svc
spec:
  clusterIP: None
  selector:
    app: redis-cluster

Create a Statefulset to deploy Redis.

When you create a StatefulSet, reference the Service that you created, mount a ConfigMap to the /config directory of each pod, create a persistent volume claim (PVC) for each pod, and then mount the PVC to the /data directory.

kubectl create -f redis.yaml

Sample content of the redis.yaml file:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
spec:
  selector:
    matchLabels:
      app: redis-cluster
  serviceName: redis-cluster-svc
  replicas: 6
  template:
    metadata:
      labels:
        app: redis-cluster
        alibabacloud.com/eci: "true"
    spec:
      terminationGracePeriodSeconds: 10
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis-cluster
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - name: redis
        image: redis:6.0.8
        command: ["redis-server", "/config/redis.conf"]
        ports:
        - name: redis
          containerPort: 6379
          protocol: TCP
        - name: election
          containerPort: 16379
          protocol: TCP
        volumeMounts:
        - name: redis-conf
          mountPath: /config
        - name: pvc-essd-redis-data
          mountPath: /data
      volumes:
      - name: redis-conf
        configMap:
          name: redis-cluster
          items:
          - key: redis.conf
            path: redis.conf
  volumeClaimTemplates:
  - metadata:
      name: pvc-essd-redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alicloud-disk-essd
      resources:
        requests:
          storage: 20Gi

After you create the Statefulset, wait until all pods in the Statefulset are in the Ready state.

kubectl get statefulset redis-cluster -o wide

The following command output is returned:

NAME            READY   AGE     CONTAINERS   IMAGES
redis-cluster   6/6     8m52s   redis        redis:6.0.8

Initialize the cluster.

Obtain the IP address of each node.

Redis does not allow you to initialize a cluster by using the hostname. You must obtain the IP address of each node when you initialize a Redis cluster.

kubectl get pods -l app=redis-cluster -o wide

The following command output is returned:

NAME              READY   STATUS    RESTARTS   AGE     IP             NODE                           NOMINATED NODE   READINESS GATES
redis-cluster-0   1/1     Running   0          9m37s   172.16.55.6    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-1   1/1     Running   0          9m5s    172.16.55.7    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-2   1/1     Running   0          8m30s   172.16.55.8    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-3   1/1     Running   0          7m46s   172.16.55.9    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-4   1/1     Running   0          7m7s    172.16.55.10   virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-5   1/1     Running   0          6m30s   172.16.55.11   virtual-kubelet-cn-beijing-k   <none>           <none>

Log on to one of the Redis nodes.

kubectl exec -ti redis-cluster-0 bash

Run the initialization command for the six nodes. Set the --cluster-replicas option to 1 to specify that each master node is assigned a slave node. This way, the cluster contains three master nodes and three slave nodes.

redis-cli -a pass123 --cluster create 172.16.55.6:6379 172.16.55.7:6379 172.16.55.8:6379 172.16.55.9:6379 172.16.55.10:6379 172.16.55.11:6379 --cluster-replicas 1

If the following command output is returned, the cluster is initialized.

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Use the Redis cluster.
After you create a headless Service, the kubernetes cluster assigns a Domain Name System (DNS) A record to the Service in the {service name}.{service namespace}.svc.{domain} format. The DNS record is mapped to the IP address of a backend pod. Each time you access the Service name, the Service name is resolved to a random Redis node.
This allows you to access the Redis cluster by accessing a pod in the cluster.
```
redis-cli -a pass123 -c -h redis-cluster-svc.default.svc.cluster.local -p 6379
```
Sample test:
```
172.16.55.8> set k1 v1
OK
172.16.55.8> get k1
"v1"
```

Scale up a Redis cluster

You cannot dynamically scale up a Redis cluster due to the following reasons: First, every time a new node is added, Redis must reallocate hash slots to all nodes in the cluster. You can add a script to the Redis image to automate the reallocation of hash slots every time a pod starts. However, when a large number of data is added to the cluster, continuous resharding slows down the scale-up of the cluster. Sharding may also exhaust the bandwidth of the Redis cluster, which may cause all clients that depend on Redis to time out. Second, no suitable policy can be used to determine whether to use a restarted pod as a master node or a slave node.

To address the preceding issues, the following example shows how to scale up a Redis cluster by performing manual sharding.

Change the number of replicas in the Statefulset from 6 to 8.
```
kubectl scale statefulsets redis-cluster --replicas=8
```

Obtain the IP addresses of the new nodes.

After all pods are in the Ready state, run the following command to obtain the IP addresses of the new nodes:

kubectl get pods -l app=redis-cluster -o wide

The following command output is returned. redis-cluster-6 and redis-cluster-7 are new nodes.

NAME              READY   STATUS    RESTARTS   AGE   IP             NODE                           NOMINATED NODE   READINESS GATES
redis-cluster-0   1/1     Running   0          88m   172.16.55.6    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-1   1/1     Running   0          88m   172.16.55.7    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-2   1/1     Running   0          87m   172.16.55.8    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-3   1/1     Running   0          87m   172.16.55.9    virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-4   1/1     Running   0          86m   172.16.55.10   virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-5   1/1     Running   0          85m   172.16.55.11   virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-6   1/1     Running   0          52s   172.16.55.16   virtual-kubelet-cn-beijing-k   <none>           <none>
redis-cluster-7   1/1     Running   0          32s   172.16.55.17   virtual-kubelet-cn-beijing-k   <none>           <none>

Add a master node.
Log on to one of the nodes. In this topic, log on to redis-cluster-0. Run the following command to add redis-cluster-6 as a master node of the cluster:
```
kubectl exec -ti redis-cluster-0 bash
```
```
redis-cli -a pass123 --cluster add-node 172.16.55.16:6379 172.16.55.6:6379
```
Note
In the preceding command, 172.16.55.6:6379 is the address of an existing node. After you connect to the node, redis-cli automatically obtains the addresses of other nodes.
Query the ID of redis-cluster-6.
```
redis-cli -a pass123 -c cluster nodes | grep 172.16.55.16:6379 | awk '{print $1}'
```
The following command output is returned:
```
47879390ecc7635f5c57d3e324e2134b24******
```
Reallocate hash slots.
After you add redis-cluster-6 as a master node to the cluster, four master nodes exist in the Redis cluster. The four nodes share 16,384 hash slots. Each master node is allocated 16,384/4 = 4,096 hash slots on average. Therefore, 4,096 hash slots must be moved from the existing three master nodes to the new master node.
Run the following command to reallocate the hash slots. 172.16.55.6:6379 is the address of an existing node. After you connect to the node, redis-cli automatically obtains the addresses of other nodes.
```
redis-cli -a pass123 --cluster reshard 172.16.55.6:6379
```
Enter the required information in sequence based on the prompts of the preceding interactive command.
```
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 47879390ecc7635f5c57d3e324e2134b24******
Source node #1: all
Do you want to proceed with the proposed reshard plan (yes/no)? yes
```
- The number of hash slots to be allocated to the new node (redis-cluster-6): 4,096
- The ID of the receiving node (the new node): 47879390ecc7635f5c57d3e324e2134b24******
- The ID of the source nodes (all existing nodes): all
- Confirm reallocation: yes
Wait until the hash slots are reallocated. During the reallocation, the Redis cluster can continue to provide services.

Add the slave node.

Run the following command to add redis-cluster-7 as a slave node of redis-cluster-6:

redis-cli -a pass123 --cluster add-node 172.16.55.17:6379 172.16.55.6:6379 --cluster-slave --cluster-master-id 47879390ecc7635f5c57d3e324e2134b24******

Note

In the preceding command, 172.16.55.6:6379 is the address of an existing node. After you connect to the node, redis-cli automatically obtains the addresses of other nodes.
47879390ecc7635f5c57d3e324e2134b24****** is the ID of the master node (redis-cluster-6).

After you add the slave node, you can see four master nodes that correspond to four slave nodes in the Redis cluster.

redis-cli -a pass123 -c cluster nodes

Sample output:

47879390ecc7635f5c57d3e324e2134b24****** 172.16.55.16:6379@16379 master - 0 1667383406000 7 connected 0-1364 5461-6826 10923-12287
75a7398a45f9696066eaa6ac7968b13a47****** 172.16.55.11:6379@16379 slave b8f6b826241b47f29a4bdde14104b28fe8****** 0 1667383406514 2 connected
b7849f8577e43d5a6da51bc78ae809bbb1****** 172.16.55.17:6379@16379 slave 47879390ecc7635f5c57d3e324e2134b24****** 0 1667383406314 7 connected
b8f6b826241b47f29a4bdde14104b28fe8****** 172.16.55.7:6379@16379 master - 0 1667383406815 2 connected 6827-10922
ffce547186e0be179830cb0dca47c203f6****** 172.16.55.6:6379@16379 myself,master - 0 1667383406000 1 connected 1365-5460
833a939cde93991c8a16c41fb2568b8642****** 172.16.55.8:6379@16379 master - 0 1667383406514 3 connected 12288-16383
a4df1a26394bfcb833914278744219db01****** 172.16.55.9:6379@16379 slave 833a939cde93991c8a16c41fb2568b8642****** 0 1667383407515 3 connected
c4234d71343ca4dbe07822e17f7572ac30****** 172.16.55.10:6379@16379 slave ffce547186e0be179830cb0dca47c203f6****** 0 1667383405813 1 connected

Scale down a Redis cluster

During scale-down of Redis clusters, StatefulSets can only delete pods one by one in the reverse order that pods were created. If you want to delete a master node and a slave node from the Redis cluster, you must use the Statefulset to delete redis-cluster-7 and redis-cluster-6. Redis must allocate all hash slots on the two nodes to other nodes before the StatefulSet deletes the nodes. Otherwise, data loss occurs, and the Redis cluster rejects services.

Reallocate hash slots.
To prevent load skew on the remaining three master nodes, the StatefulSet must evenly allocate the hash slots of redis-cluster-6 to the remaining master nodes. In this case, Redis must perform three resharding operations. The procedure is similar to the scale-up procedure. However, in each resharding operation, you must set the receiving node to one of the remaining master nodes, and the source node to redis-cluster-6. Sample command:
```
redis-cli -a pass123 --cluster reshard 172.16.55.6:6379
```
Enter the required information in sequence based on the prompts of the preceding interactive command.
```
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? ffce547186e0be179830cb0dca47c203f6******
Source node #1: 47879390ecc7635f5c57d3e324e2134b24******
Source node #2: done
Do you want to proceed with the proposed reshard plan (yes/no)? yes
```
- The number of hash slots to be allocated to the remaining master nodes: 4,096
- The ID of the receiving node (one of the remaining nodes): ffce547186e0be179830cb0dca47c203f6******
- The ID of the source node (redis-cluster-6): 47879390ecc7635f5c57d3e324e2134b24******
- Confirm reallocation: yes
Change the number of replicas in the Statefulset from 8 to 6.
```
kubectl scale statefulsets redis-cluster --replicas=6
```
View the pod information of the Redis cluster. You can see that the number of nodes in the cluster is reduced to six.
```
kubectl get pods -l app=redis-cluster -o wide
```

Delete a Redis cluster

To delete a Redis cluster, you must delete the StatefulSet, Service, and ConfigMap.

kubectl delete statefulset redis-cluster
kubectl delete svc redis-cluster-svc
kubectl delete cm redis-cluster

After you delete the StatefulSet, the related PVC is not automatically deleted. You must manually delete the PVC. After you delete the PVC, the corresponding disks and data are also deleted.

kubectl delete pvc -l app=redis-cluster