High-availability Redis cluster in Docker

Recently I have seen some systems encountering unavailability of some services due to Redis service failure. So I hope to establish a Redis cluster image to manage the scattered Redis instances in a unified way to secure high availability and automatic failover.


I. Redis cluster category
We know that there are two categories of Redis clusters: one is Redis Sentinel, a high-availability cluster with only one master database at the same time and data on various instances is consistent; the other is Redis Cluster, a distributed cluster with multiple master databases at the same time and data is sliced and deployed on various master instances. Based on our demand and Redis technology readiness, this time we will establish a Redis Sentinel cluster.
Introduction:
Redis Sentinel is used to manage multiple Redis instances. The system executes the follow three tasks:
• Monitoring: Sentinel will continually check whether your master instance and slave instance work normally.
• Notification: when a monitored Redis instance encounters some problems, Sentinel can issue notifications through API to the administrator or other applications.
• Automatic failover: when a master instance fails, Sentinel will automatically initiate the failover operation. It will promote a slave instance of the failed master instance to the new master instance, and order other slave instances of the failed master instance to replicate from the new master instance. When the client attempts to connect to the failed master instance, the cluster will return the address of the new master instance, so that the new master instance can replace the failed instance.
II. Image production
The entire cluster can be divided into one master instance, N slave instances and M sentinel instances. In this example, we suppose there are 2 slave instances and 3 sentinel instances:


First, add redis.conf
##redis.conf
##redis-0, master by default
port $redis_port
##Authorization password. Please use consistent configurations.
##Temporarily disable command rename
##rename-command
##Enable AOF, disable snapshot
appendonly yes
#slaveof redis-master $master_port
slave-read-only yes

Master by default. After #slaveof is commented out, it turns slave. Here the master domain name is fixed as redis-master.
Add sentinel.conf
port $sentinel_port
dir "/tmp"
##The Redis name, IP address and port monitored by Sentinel. The last number is the smallest number of sentinel instances required for voting at Sentinel decision-making.
sentinel monitor mymaster redis-master $master_port 2
##This option designates the maximum number of slave instances allowed for synchronization with the new master instance at the same time during a failover. The smaller the number, the longer the time needed to complete the failover.
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1

Add the startup script. Determine the startup of master, slave and sentinel instances according to the incoming arguments.
cd /data
redis_role=$1
echo $redis_role
if [ $redis_role = "master" ] ; then
    echo "master"
    sed -i "s/\$redis_port/$redis_port/g" redis.conf
    redis-server /data/redis.conf
elif [ $redis_role = "slave" ] ; then  
    echo "slave"
    sed -i "s/\$redis_port/$redis_port/g" redis.conf
    sed -i "s/#slaveof/slaveof/g" redis.conf
    sed -i "s/\$master_port/$master_port/g" redis.conf
    redis-server /data/redis.conf
elif [ $redis_role = "sentinel" ] ; then
    echo "sentinel"
    sed -i "s/\$sentinel_port/$sentinel_port/g" sentinel.conf
    sed -i "s/\$master_port/$master_port/g" sentinel.conf
    redis-sentinel /data/sentinel.conf
else
    echo "unknow role!"
fi     #ifend

Among them, the $redis_port, $master_port and $sentinel_port are all from the environment variables and passed in at Docker startup.
Write Dockerfile
FROM redis:3-alpine
MAINTAINER voidman <voidman>

COPY Shanghai /etc/localtime
COPY redis.conf /data/redis.conf
COPY sentinel.conf /data/sentinel.conf
COPY start.sh /data/start.sh
RUN chmod +x /data/start.sh
RUN chown redis:redis /data/*
ENTRYPOINT ["sh","/data/start.sh"]
CMD ["master"]

Select redis-alpine image as the basic image, because it is very small, only 9M. After you modify the time zone and copy some configurations, change the permission and user group, because the basic image is of the Redis user group. Use ENTRYPOINT and CMD functions in combination, master startup by default.
After image build completes, the image is only 15M.
III. Startup
Use docker-compose format:
redis-master-host:
  environment:
    redis_port: '16379'
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: xxx.aliyun.com:5000/aegis-redis-ha:1.0
  stdin_open: true
  net: host
redis-slaves:
  environment:
    master_port: '16379'
    redis_port: '16380'
  labels:
    io.rancher.scheduler.affinity:container_label_soft_ne: name=slaves
    io.rancher.container.pull_image: always
    name: slaves
  tty: true
  command:
  - slave
  image: xxx.aliyun.com:5000/aegis-redis-cluster:1.0
  stdin_open: true
  net: host
redis-sentinels:
  environment:
    master_port: '16379'
    sentinel_port: '16381'
  labels:
    io.rancher.container.pull_image: always
    name: sentinels
    io.rancher.scheduler.affinity:container_label_ne: name=sentinels
  tty: true
  command:
  - sentinel
  image: xxx.aliyun.com:5000/aegis-redis-cluster:1.0
  stdin_open: true
  net: host

First, start the master instance, pass in port 16379, in host mode. Then start slave instances for 16379 master instance, and set the scheduling policy as scattered as possible. Follow the same steps for sentinels.
IV. Test
Java client test (segment):
//Initialization
 Set<String> sentinels = new HashSet<String>(16);
            sentinels.add("redis-sentinel1.aliyun.com:16381");
            sentinels.add("redis-sentinel2.aliyun.com:16381");
            sentinels.add("redis-sentinel3.aliyun.com:16381");
            GenericObjectPoolConfig config = new GenericObjectPoolConfig();
            config.setBlockWhenExhausted(true);
            config.setMaxTotal(10);
            config.setMaxWaitMillis(1000l);
            config.setMaxIdle(25);
            config.setMaxTotal(32);
            jedisPool = new JedisSentinelPool("mymaster", sentinels, config);
//Read and write endlessly
 while (true) {
                AegisRedis.set("testSentinel", "ok");
                System.err.println(AegisRedis.get("testSentinel"));
                Thread.sleep(3000);
        }

Sentinel instance failure test
Kill a sentinel instance, and the following prompt will display:
Severe:  Lost connection to Sentinel at redis-sentinel2.aliyun.com:16381. Sleeping 5000ms and retrying.

Data can be read and written normally. When all the sentinel instances are killed, data read and write are normal, with repeated attempts to connect to sentinel instances. This indicates that the sentinel instance is only useful during revote for a new master instance and failover. Once the master instance is determined, Redis can work normally even when all the sentinel instances fail.
However, if this happens during redisPool re-initialization, an error will be reported:
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is mymaster master is running...

No configurations are required between various sentinel instances, as they all subscribe to the sentinel:hello channel of the master and slave instances. By reporting its own IP addresses and ports, each sentinel instance maintains a known sentinel list.
Slave instance failure test
Kill a slave instance and it generates no impact to the client which will not perceive it. The master instance, however, will log the loss of connection:
2016/4/14 16:31:336:M 14 Apr 16:31:33.698 # Connection with slave ip_address:16380 lost.

The sentinel instances also have logs:
2016/4/14 16:30:397:X 14 Apr 16:30:39.852 # -sdown slave ip_address:16380 ip_address 16380 @ mymaster ip_address 16379
2016/4/14 16:32:037:X 14 Apr 16:32:03.786 # +sdown slave ip_address:16380 ip_address 16380 @ mymaster ip_address 16379

Restore the slave instance
2016/4/14 16:36:579:S 14 Apr 16:36:57.441 * Connecting to MASTER redis-master:16379
2016/4/14 16:36:579:S 14 Apr 16:36:57.449 * MASTER <-> SLAVE sync started
2016/4/14 16:36:579:S 14 Apr 16:36:57.449 * Non blocking connect for SYNC fired the event.
2016/4/14 16:36:579:S 14 Apr 16:36:57.449 * Master replied to PING, replication can continue...
2016/4/14 16:36:579:S 14 Apr 16:36:57.449 * Partial resynchronization not possible (no cached master)
2016/4/14 16:36:579:S 14 Apr 16:36:57.450 * Full resync from master: 0505a8e1049095ce597a137ae1161ed4727533d3:84558
2016/4/14 16:36:579:S 14 Apr 16:36:57.462 * SLAVE OF ip_address:16379 enabled (user request from 'id=3 addr=ip_address2:57122 fd=10 name=sentinel-11d82028-cmd age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=rw cmd=exec')
2016/4/14 16:36:579:S 14 Apr 16:36:57.462 # CONFIG REWRITE executed with success.
2016/4/14 16:36:589:S 14 Apr 16:36:58.451 * Connecting to MASTER ip_address:16379
2016/4/14 16:36:589:S 14 Apr 16:36:58.451 * MASTER <-> SLAVE sync started
2016/4/14 16:36:589:S 14 Apr 16:36:58.451 * Non blocking connect for SYNC fired the event.
2016/4/14 16:36:589:S 14 Apr 16:36:58.451 * Master replied to PING, replication can continue...
2016/4/14 16:36:589:S 14 Apr 16:36:58.451 * Partial resynchronization not possible (no cached master)
2016/4/14 16:36:589:S 14 Apr 16:36:58.453 * Full resync from master: 0505a8e1049095ce597a137ae1161ed4727533d3:84721
2016/4/14 16:36:589:S 14 Apr 16:36:58.532 * MASTER <-> SLAVE sync: receiving 487 bytes from master
2016/4/14 16:36:589:S 14 Apr 16:36:58.532 * MASTER <-> SLAVE sync: Flushing old data
2016/4/14 16:36:589:S 14 Apr 16:36:58.532 * MASTER <-> SLAVE sync: Loading DB in memory
2016/4/14 16:36:589:S 14 Apr 16:36:58.532 * MASTER <-> SLAVE sync: Finished with success
2016/4/14 16:36:589:S 14 Apr 16:36:58.537 * Background append only file rewriting started by pid 12
2016/4/14 16:36:589:S 14 Apr 16:36:58.563 * AOF rewrite child asks to stop sending diffs.
2016/4/14 16:36:5812:C 14 Apr 16:36:58.563 * Parent agreed to stop sending diffs. Finalizing AOF...
2016/4/14 16:36:5812:C 14 Apr 16:36:58.563 * Concatenating 0.00 MB of AOF diff received from parent.
2016/4/14 16:36:5812:C 14 Apr 16:36:58.563 * SYNC append only file rewrite performed
2016/4/14 16:36:5812:C 14 Apr 16:36:58.564 * AOF rewrite: 0 MB of memory used by copy-on-write
2016/4/14 16:36:589:S 14 Apr 16:36:58.652 * Background AOF rewrite terminated with success
2016/4/14 16:36:589:S 14 Apr 16:36:58.653 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
2016/4/14 16:36:589:S 14 Apr 16:36:58.653 * Background AOF rewrite finished successfully

It will immediately recover data from the master instance until the data is consistent.
Master instance failure test
The client shows exceptions:
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused

The sentinel instance first discovers the issue, and makes a subjective judgment that the master instance (ip_address 16379) failed. It then queries other sentinel instances to confirm the judgment. After it receives confirmation from 2 sentinel instances (the number 2 here is previously configured in sentinel.conf. The number is usually recommended to be more than half of the number of sentinel instances), it makes an objective judgment that the master instance has failed. They start a new round of vote for a new master instance, and master (ip_address:16380) is elected. So failover starts, and the cluster is switched to the new master instance. Other slave instances are notified about the new master instance. Below are the detailed logs: it is worth noting that during the selection of a new master instance, a short interruption occurs to the client side.
2016/4/14 16:40:3613:X 14 Apr 16:40:36.162 # +sdown master mymaster ip_address 16379
2016/4/14 16:40:3613:X 14 Apr 16:40:36.233 # +odown master mymaster ip_address 16379 #quorum 2/2
2016/4/14 16:40:3613:X 14 Apr 16:40:36.233 # +new-epoch 10
2016/4/14 16:40:3613:X 14 Apr 16:40:36.233 # +try-failover master mymaster ip_address 16379
2016/4/14 16:40:3613:X 14 Apr 16:40:36.238 # +vote-for-leader 0a632ec0550401e66486846b521ad2de8c345695 10
2016/4/14 16:40:3613:X 14 Apr 16:40:36.249 # ip_address2:16381 voted for 0a632ec0550401e66486846b521ad2de8c345695 10
2016/4/14 16:40:3613:X 14 Apr 16:40:36.261 # ip_address3:16381 voted for 4e590c09819a793faf1abf185a0d0db07dc89f6a 10
2016/4/14 16:40:3613:X 14 Apr 16:40:36.309 # +elected-leader master mymaster ip_address 16379
2016/4/14 16:40:3613:X 14 Apr 16:40:36.309 # +failover-state-select-slave master mymaster ip_address 16379
2016/4/14 16:40:3613:X 14 Apr 16:40:36.376 # +selected-slave slave ip_address:16380 ip_address 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3613:X 14 Apr 16:40:36.376 * +failover-state-send-slaveof-noone slave ip_address:16380 ip_address 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3613:X 14 Apr 16:40:36.459 * +failover-state-wait-promotion slave ip_address:16380 ip_address 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3713:X 14 Apr 16:40:37.256 # +promoted-slave slave ip_address:16380 ip_address 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3713:X 14 Apr 16:40:37.256 # +failover-state-reconf-slaves master mymaster ip_address 16379
2016/4/14 16:40:3713:X 14 Apr 16:40:37.303 * +slave-reconf-sent slave ip_address3:16380 ip_address3 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3813:X 14 Apr 16:40:38.288 * +slave-reconf-inprog slave ip_address3:16380 ip_address3 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3813:X 14 Apr 16:40:38.289 * +slave-reconf-done slave ip_address3:16380 ip_address3 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3813:X 14 Apr 16:40:38.378 * +slave-reconf-sent slave ip_address2:16380 ip_address2 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3813:X 14 Apr 16:40:38.436 # -odown master mymaster ip_address 16379
2016/4/14 16:40:3913:X 14 Apr 16:40:39.368 * +slave-reconf-inprog slave ip_address2:16380 ip_address2 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3913:X 14 Apr 16:40:39.368 * +slave-reconf-done slave ip_address2:16380 ip_address2 16380 @ mymaster ip_address 16379
2016/4/14 16:40:3913:X 14 Apr 16:40:39.424 # +failover-end master mymaster ip_address 16379
2016/4/14 16:40:3913:X 14 Apr 16:40:39.424 # +switch-master mymaster ip_address 16379 ip_address 16380
2016/4/14 16:40:3913:X 14 Apr 16:40:39.425 * +slave slave ip_address3:16380 ip_address3 16380 @ mymaster ip_address 16380
2016/4/14 16:40:3913:X 14 Apr 16:40:39.425 * +slave slave ip_address2:16380 ip_address2 16380 @ mymaster ip_address 16380
2016/4/14 16:40:3913:X 14 Apr 16:40:39.425 * +slave slave ip_address:16379 ip_address 16379 @ mymaster ip_address 16380

If the old master instance recovers at this time and finds it has been defined as a slave instance of the new master instance by the sentinels, it has no other choice but to obey the arrangement and synchronize data from the master instance to keep data consistency.
V. Summary
In general, as long as one Redis instance stays alive in the cluster, the cluster is able to provide external services, and the sentinel instance is only working when the master or slave instances fail.
The image in this example is only 15M. Roles and ports are configured at startup, including 3 roles, namely master, slave and sentinel, and they are arranged through services to start a Redis cluster.