{"id":6288,"date":"2021-11-15T21:21:12","date_gmt":"2021-11-15T21:21:12","guid":{"rendered":"https:\/\/www.geekdecoder.com\/?p=6288"},"modified":"2021-11-15T21:21:12","modified_gmt":"2021-11-15T21:21:12","slug":"install-redis-high-availability-architecture-with-sentinel","status":"publish","type":"post","link":"https:\/\/www.qbytes.cloud\/index.php\/2021\/11\/15\/install-redis-high-availability-architecture-with-sentinel\/","title":{"rendered":"Install Redis High Availability Architecture with Sentinel"},"content":{"rendered":"\n

Redis Sentinel is a dedicated process to automate and simplify the Redis replication failover and switchover.<\/p>\n\n\n\n

In this article, we will start with 3 Nodes. We will have two Redis instances on two different nodes – 1 master and 1 replica (or slave). Sentinel will be co-located on those 2 nodes, plus an additional node on one of our web servers.<\/p>\n\n\n\n

Normally you would co-locate a Redis instance on the web\/application server and access it via localhost or through a UNIX socket file. This is the straightforward way to incorporate Redis into the application.<\/p>\n\n\n\n\n\n\n\n

For a scalable and highly available setup app, Redis should be deployed in a centralized approach, or into a different tier called a cache tier. This allows Redis instances to work together as a dedicated cache provider for the applications, decoupling the applications from the local Redis dependencies.<\/p>\n\n\n\n

For this deployment we are setting up 3 Debian 11 VM instances.<\/p>\n\n\n\n

Before deploying Redis Sentinel, we have to deploy a Redis replication consisting of two or more Redis Server instances. Let’s start by installing Redis on both servers, Redis-01 and redis-02:<\/p>\n\n\n\n

1 – 192.168.0.30 – App- Web Server + Sentinel
2 – 192.168.0.31 – Redis-01 (replica) Redis Server + Sentinel
3 – 192.168.0.32 – Redis-02 (replica) Redis Server + Sentinel<\/p>\n\n\n

\napt install redis net-tools\n<\/pre><\/div>\n\n\n
Edit \/etc\/redis\/redis.conf:<\/p>\n\n\n\n
For redis-01 (master):<\/p>\n\n\n
\nbind 127.0.0.1 192.168.0.31\nprotected-mode no\nsupervised systemd\nmasterauth Zssy56G21Zx\nmasteruser redisuser\nuser redisuser +@all on >Zssy56G21Zx\n<\/pre><\/div>\n\n\n
For redis-02 (replica):<\/p>\n\n\n
\nbind 127.0.0.1 192.168.0.32\nprotected-mode no\nsupervised systemd\nreplicaof 192.168.0.31 6379\nmasterauth Zssy56G21Zx\nmasteruser redisuser\nuser redisuser +@all on >Zssy56G21Zx\n<\/pre><\/div>\n\n\n
    \n
  • bind: List all the IP addresses that you want Redis to listen to. For Sentinel to work properly, Redis must be reachable remotely. Therefore we have to list out the interface the Sentinel will communicate with.<\/li>\n\n\n\n
  • protected-mode: This must be set to “no” to allow Redis to serve remote connections. This is required for Sentinel as well.<\/li>\n\n\n\n
  • supervised: We use the default systemd unit files provided by the installer package. For Ubuntu 20.04, it uses systemd as the service manager so we specify systemd here.<\/li>\n\n\n\n
  • replicaof: This is only for the slave node. For the original topology, we will make redis2 as the replica and redis1 as the master.<\/li>\n\n\n\n
  • masterauth: The password for user masteruser.<\/li>\n\n\n\n
  • masteruser: The username of the master user.<\/li>\n\n\n\n
  • user: We create the master user here. The user shall have no limit (+@all) and a password. This user will be used for Redis to manage replication and failover by Sentinel.<\/li>\n<\/ul>\n\n\n\n
    Restart Redis to apply the changes and enable it on boot:<\/p>\n\n\n
    \nsudo systemctl enable redis-server\n<\/pre><\/div>\n\n\n
    Verify that Redis is running on port 6379 on both interfaces. The following example is the output from redis2:<\/p>\n\n\n
    \nsudo netstat -tulpn | grep -i redis\ntcp        0      0 192.168.0.32:6379       0.0.0.0:*               LISTEN      1246\/redis-server 1\ntcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      1246\/redis-server 1\ntcp        0      0 127.0.0.1:26379         0.0.0.0:*               LISTEN      1128\/redis-sentinel\ntcp6       0      0 ::1:26379               :::*                    LISTEN      1128\/redis-sentinel\n<\/pre><\/div>\n\n\n
    Verify the replication is working. On redis-01:<\/p>\n\n\n
    \nredis-cli info replication\n<\/pre><\/div>\n\n\n
    Output:<\/p>\n\n\n
    \nReplication\nrole:master\nconnected_slaves:1\nslave0:ip=192.168.0.32,port=6379,state=online,offset=10621,lag=1\nmaster_replid:c649cbc9ffab2b6ef7adae7df90a1d5d00913987\nmaster_replid2:0000000000000000000000000000000000000000\nmaster_repl_offset:10754\nsecond_repl_offset:-1\nrepl_backlog_active:1\nrepl_backlog_size:1048576\nrepl_backlog_first_byte_offset:1\nrepl_backlog_histlen:10754\n<\/pre><\/div>\n\n\n

    Pay attention to the role, connected_slaves and slave{i} keys. This indicates that redis1 is the master. Also, note that a replica can be a master of another replica – this is also known as chained replication.<\/p>\n\n\n\n
    While on the redis2:<\/p>\n\n\n
    \nredis-cli info replication\n<\/pre><\/div>\n\n\n
    Output:<\/p>\n\n\n
    \nReplication\nrole:slave\nmaster_host:192.168.0.31\nmaster_port:6379\nmaster_link_status:up\nmaster_last_io_seconds_ago:1\nmaster_sync_in_progress:0\nslave_repl_offset:52453\nslave_priority:100\nslave_read_only:1\nconnected_slaves:0\nmaster_replid:c649cbc9ffab2b6ef7adae7df90a1d5d00913987\nmaster_replid2:0000000000000000000000000000000000000000\nmaster_repl_offset:52453\nsecond_repl_offset:-1\nrepl_backlog_active:1\nrepl_backlog_size:1048576\nrepl_backlog_first_byte_offset:1\nrepl_backlog_histlen:52453\n<\/pre><\/div>\n\n\n
    Pay attention to the role, master_host, master_link_status and master_repl_offset. The replication delay between these two nodes can be determined by the master_repl_offset value on both servers.<\/p>\n\n\n\n
    Redis Sentinel Deployment<\/h3>\n\n\n\n
    Redis Sentinel is basically the same redis-server process running with the “–sentinel” flag and different configuration files and ports. For production usage, it is strongly recommended to have at least 3 Sentinel instances for an accurate observation when performing the automatic failover. Therefore, we will install Sentinel on those 2 Redis nodes that we have, plus one of our web servers, 192.168.44.70 (shown in the architecture diagram).<\/p>\n\n\n\n
    Install the redis-sentinel package on the selected web server (Sentinel is already installed on our Redis hosts):<\/p>\n\n\n
    \napt install redis-sentinel\n<\/pre><\/div>\n\n\n
    Stop redis sentinel to make changes<\/p>\n\n\n
    \nsystemctl stop redis-sentinel.service\n<\/pre><\/div>\n\n\n
    By default, the Sentinel configuration file is located at \/etc\/redis\/sentinel.conf. Make sure the following configuration lines are set:<\/p>\n\n\n\n
    App server, 192.168.0.30:<\/p>\n\n\n
    \nbind 192.168.0.30\nport 26379\nsentinel monitor mymaster 192.168.0.31 6379 2\nsentinel auth-pass mymaster Zssy56G21Zx\nsentinel auth-user mymaster redisuser\nsentinel down-after-milliseconds mymaster 10000\n<\/pre><\/div>\n\n\n

    redis-01, 192.168.0.31:<\/p>\n\n\n
    \nbind 192.168.0.31\nport 26379\nsentinel monitor mymaster 192.168.0.31 6379 2\nsentinel auth-pass mymaster Zssy56G21Zx\nsentinel auth-user mymaster redisuser\nsentinel down-after-milliseconds mymaster 10000\n<\/pre><\/div>\n\n\n
    redis-02, 192.168.0.32:<\/p>\n\n\n
    \nbind 192.168.0.32\nport 26379\nsentinel monitor mymaster 192.168.0.31 6379 2\nsentinel auth-pass mymaster Zssy56G21Zx\nsentinel auth-user mymaster redisuser\nsentinel down-after-milliseconds mymaster 10000\n<\/pre><\/div>\n\n\n
    Restart the redis-sentinel daemon to apply the changes:<\/p>\n\n\n
    \nsudo systemctl restart redis-sentinel\nsudo systemctl enable redis-sentinel\n<\/pre><\/div>\n\n\n
    Make sure redis-sentinel is running on port 26379. On redis2, you should see something like this:<\/p>\n\n\n
    \nnetstat -tupln | grep redis\n<\/pre><\/div>\n\n\n
    Output:<\/p>\n\n\n
    \ntcp        0      0 192.168.0.31:26379      0.0.0.0:*               LISTEN      1170\/redis-sentinel\ntcp        0      0 192.168.0.31:6379       0.0.0.0:*               LISTEN      881\/redis-server 12\ntcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      881\/redis-server 12\n<\/pre><\/div>\n\n\n
    Verify if the Sentinel is observing our Redis replication link by looking at the log file, \/var\/log\/redis\/redis-sentinel.log. Make sure you see the following lines:<\/p>\n\n\n
    \ntail -f \/var\/log\/redis\/redis-sentinel.log\n<\/pre><\/div>\n\n\n
    Output:<\/p>\n\n\n
    \n553:X 01 Dec 2021 11:55:36.378 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.\n553:X 01 Dec 2021 11:55:36.379 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\n553:X 01 Dec 2021 11:55:36.379 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=553, just started\n553:X 01 Dec 2021 11:55:36.379 # Configuration loaded\n553:X 01 Dec 2021 11:55:36.380 * Running mode=sentinel, port=26379.\n553:X 01 Dec 2021 11:55:36.380 # Sentinel ID is 1fc2ddf0a56cb12f8cfe91da98d643fde80c7f29\n553:X 01 Dec 2021 11:55:36.380 # +monitor master mymaster 192.168.0.31 6379 quorum 2\n553:X 01 Dec 2021 11:55:37.399 * +sentinel sentinel 1cf43ff56ead25a832eb0f5ecd9a1e9ae4cdbbc1 192.168.0.31 26379 @ mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 11:55:37.525 # +new-epoch 3\n553:X 01 Dec 2021 11:55:38.084 * +sentinel sentinel 5d27f1340e5a04dc6703c60d7b60e7c6a121b44c 192.168.0.32 26379 @ mymaster 192.168.0.31 6379\n<\/pre><\/div>\n\n\n
    We can get more information on the Sentinel process by using the redis-cli and connect to the Sentinel port 26379. From the app server<\/strong>, run:<\/p>\n\n\n
    \nredis-cli -h 192.168.0.32 -p 26379 sentinel masters\n<\/pre><\/div>\n\n\n
    Output:<\/p>\n\n\n
    \n    1) "name"\n    2) "mymaster"\n    3) "ip"\n    4) "192.168.0.31"\n    5) "port"\n    6) "6379"\n    7) "runid"\n    8) "ebad0adccca0d81eb09c119b5a44fd5043994d11"\n    9) "flags"\n   10) "master"\n   11) "link-pending-commands"\n   12) "0"\n   13) "link-refcount"\n   14) "1"\n   15) "last-ping-sent"\n   16) "0"\n   17) "last-ok-ping-reply"\n   18) "274"\n   19) "last-ping-reply"\n   20) "274"\n   21) "down-after-milliseconds"\n   22) "30000"\n   23) "info-refresh"\n   24) "4035"\n   25) "role-reported"\n   26) "master"\n   27) "role-reported-time"\n   28) "1038344"\n   29) "config-epoch"\n   30) "0"\n   31) "num-slaves"\n   32) "1"\n   33) "num-other-sentinels"\n   34) "2"\n   35) "quorum"\n   36) "2"\n   37) "failover-timeout"\n   38) "180000"\n   39) "parallel-syncs"\n   40) "1"\n<\/pre><\/div>\n\n\n
    Check the “num-slaves” value which is 1 and “num-other-sentinels” value which is 2, indicating that we have a total number of 3 Sentinel nodes (one for this node + two other nodes).
    Failover Testing<\/p>\n\n\n\n
    We can now test the failover by simply shutting down the Redis service on redis1(master):<\/p>\n\n\n
    \nsudo systemctl stop redis-server\n<\/pre><\/div>\n\n\n
    After 10 seconds (down-after-milliseconds value), you should see the following output in the \/var\/log\/redis\/redis-sentinel.log file:<\/p>\n\n\n
    \ntail -f \/var\/log\/redis\/redis-sentinel.log\n<\/pre><\/div>\n\n\n
    Output<\/p>\n\n\n
    \n553:X 01 Dec 2021 12:41:31.440 # +elected-leader master mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:31.440 # +failover-state-select-slave master mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:31.523 # +selected-slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:31.523 * +failover-state-send-slaveof-noone slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:31.586 * +failover-state-wait-promotion slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:32.371 # +promoted-slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:32.371 # +failover-state-reconf-slaves master mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:32.413 # +failover-end master mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:41:32.413 # +switch-master mymaster 192.168.0.31 6379 192.168.0.32 6379\n553:X 01 Dec 2021 12:41:32.413 * +slave slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379\n553:X 01 Dec 2021 12:41:42.454 # +sdown slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379\n553:X 01 Dec 2021 12:43:17.878 # -sdown slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379\n<\/pre><\/div>\n\n\n
    Now, the slave, 192.168.0.32, has been promoted to a master. Once our old master (redis-01) comes back online, you should see something like this reported by Sentinel:<\/p>\n\n\n
    \n553:X 01 Dec 2021 12:41:32.413 * +slave slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379\n<\/pre><\/div>\n\n\n
    The above indicates the old master has been converted to slave and now replicating from the current master, redis2. We can confirm this by checking the replication info on redis2:<\/p>\n\n\n
    \nredis-cli info replication\n<\/pre><\/div>\n\n\n
    Output<\/p>\n\n\n
    \nReplication\nrole:master\nconnected_slaves:0\nmaster_replid:ddf2e69e46513233a51b835babdb18823d6b4a7b\nmaster_replid2:e30c6877b48ef5408cb3f1212d040a271d4d3e92\nmaster_repl_offset:626689\nsecond_repl_offset:611794\nrepl_backlog_active:1\nrepl_backlog_size:1048576\nrepl_backlog_first_byte_offset:1\nrepl_backlog_histlen:626689\n<\/pre><\/div>\n\n\n
    If we want to promote redis1 to a master status again, we can simply bring down redis2 or use the Sentinel failover command as below:<\/p>\n\n\n
    \nredis-cli -h 192.168.0.30 -p 26379 sentinel failover mymaster\n<\/pre><\/div>\n\n\n
    Output:<\/p>\n\n\n
    \n553:X 01 Dec 2021 12:54:03.444 # +switch-master mymaster 192.168.0.32 6379 192.168.0.31 6379\n553:X 01 Dec 2021 12:54:03.444 * +slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379\n553:X 01 Dec 2021 12:54:13.653 * +convert-to-slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379\n<\/pre><\/div>\n\n\n
    Check:<\/p>\n\n\n\n
    redis-01<\/p>\n\n\n
    \nredis-cli info replication\n<\/pre><\/div>\n\n\n
    Output<\/p>\n\n\n
    \nReplication\nrole:master\nconnected_slaves:1\nslave0:ip=192.168.0.32,port=6379,state=online,offset=795039,lag=1\nmaster_replid:2b5a609a0d28227480c9126d95a915a870002811\nmaster_replid2:ddf2e69e46513233a51b835babdb18823d6b4a7b\nmaster_repl_offset:795039\nsecond_repl_offset:766456\nrepl_backlog_active:1\nrepl_backlog_size:1048576\nrepl_backlog_first_byte_offset:635864\nrepl_backlog_histlen:159176\n<\/pre><\/div>","protected":false},"excerpt":{"rendered":"
    Redis Sentinel is a dedicated process to automate and simplify the Redis replication failover and switchover. In this article, we will start with 3 Nodes. We will have two Redis instances on two different nodes – 1 master and 1 replica (or slave). Sentinel will be co-located on those 2 nodes, plus an additional node … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[95],"tags":[],"class_list":["post-6288","post","type-post","status-publish","format-standard","hentry","category-redis"],"_links":{"self":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts\/6288","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/comments?post=6288"}],"version-history":[{"count":0,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts\/6288\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/media?parent=6288"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/categories?post=6288"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/tags?post=6288"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}