So in this session we are going to talk about Redis High Availability and Redis High Availability
is different
than, you know, Redis Replication. In Redis Replication, what we have done is we had
a Master Server, then we had N number of,you know, Slave Nodes and whenever Master goes down, you can
connect to Slave Node to perform Read Only Operations.
However, if you want to convert a Slave Node to a Master Node, you have to take some sort of Manual
Action.
Now with the help of something called Redis Sentinel. This is a software, which is bundled with the Redis
Software Package.
So Redis Sentinel is going to allow us, you know, or going to provide us a Mechanism using which
we are going to Automate basically the process of making a Slave Node to a Master Node. Whenever the existing
Master Node is down.
So now what is the requirement, which you need to have in order to, you know, perform this
Redis High Availability.
So one of the key things which you need to note is, that you need to install a Sentinel on minimum of
3 Servers. okay.
Now the reason, why you need 3 servers on which you need to install Redis Sentinel is, that there
is something called Quorum.
Okay so Quorum Configuration basically is the minimum number of Sentinel Servers. Which needs to agree
that a Master Node is down and then it is going to convert one of the Slave Nodes to a Master Node.
Okay.
So Quorum, basically Redis Sentinel is going to have an Election process and in that Election process, you
know, majority of Redis Sentinels are going to decide that Yes Master Node is down and it is not
reachable.
And now this is the time they need to be promoting one of the Slave Nodes to a Master Node. Okay.
So whatever, you know, number you provide in the Quorum Configuration that is a minimum number of Servers, Sentinel
Servers or Sentinel Services, which is going to be a agree and going to perform this Election of a new
Master Server.
The other thing to notice is, that Sentinel runs on a default Port of 2 6 3 7 9. Okay whereas Redis
runs on a default Port of 6 3 7 9.
Both of these ports can be changed from the Configuration File.
So if we remember, Redis configuration is stored in Redis.conf File and Sentinel configuration is stored
in Sentinel.conf File. Now both of these files are kept at the same Directory and also there is a called
Redis-sentinel utility, which is also stored in the same Directory, where Redis-server
Utility is available. So you do not need a separate, you know, downloading and installing of this Sentinel Software.
It comes bundled with the same software.
The only thing is,you need to start the services, after doing the Correct Configuration.
Now let's discuss about the working of these Nodes. So now this Redis Sentinel, we are saying that
we need to install it on minimum of three Nodes but those Nodes do not need to be a completely separate
server. Okay that can be a same Redis Server, on which you can install Redis Sentinels.
So, in this example you can see we have a Master Node and then we have 2 Slave Nodes, which is S 1 and
S 2. Okay and then we have Redis Sentinel 1, which is running on Master node, then we have Redis Sentinel
2, which is running on Slave Node 1 and then we have Redis Sentinel 3, which is running on Slave Node 2.
Okay.
And in this case whenever Master is going to get down. In that case you can see Redis Sentinel are
going to decide, that it is the time they need to elect and promote one of the Slave Nodes to a Master.
And in this case they have promoted Slave Node to a Master Node.
Okay.
And also when one of the Slave Node is converted to a Master Node, all the previous or any previous Master
Node is going to be automatically changed to a Slave Node.
So whenever this Master Node which is M 1, comes back this is actually going to become a Slave Node.
Okay and this is going to become a Slave Node of this server, newly elected Master Server. So both of these
are basically a Slave Node of
This Master Server.
Now there is something called down after milliseconds parameter. Okay which you see here and this is the configuration
which is done in Sentinel.conf file.
So down after milliseconds number tells Sentinel that, this is the number of milliseconds Sentinel should
wait after master is not able to reach.
Okay so whatever number you define here, that is a minimum number of time, Sentinel it is going to wait for
the master to come back up.
If master is not able to connect for that number of time, then it is just going to start the Election
of a New Master Process.
In our example, we are going to you know, discuss about the same scenario. We are going to have 3
servers. On which we are going to make the same setup and we will see how all of these works.
OK.
So now let's discuss about what are the different kind of setup, which you can have and what are the, you
know some of the advantages or disadvantages of, all of those kind of scenarios.
Let's try to see what are the issues which we can have and what is a mitigation
Now the first deployment type which is shown here, is a Deployment where we have only 2 Redis Sentinel
running. Redis Sentinel 1 and Redis Sentinel 2.
So this is the Deployment which you should never do. It is never going to work.
The reason being is because Quorum is set to 2 and whenever this, you know, whenever let's say we have a problem
with this server.
And in that case and M 1 and RS 1 both these services are going to get impacted. Both of these will
not be available.
So now a Redis Sentinel 2, doesn't have the minimum number of Redis sentinels available, to perform
the Election process and convert this Slave Node to a Master Node. So that's the reason it is recommended
that you never do this Deployment
okay.
The only way when this setup is going to work is, when there is, you know, the server on which Redis Sentinel
1 is running.
Okay.
That server somehow is available and Redis Sentinel 1 is also available and there is some impact
only on the Redis Master Node.
Let's say for some reason Redis Master Node, you know, service is not running but that server is available
and Redis Sentinel 1 is running and Redis Sentinel 2 is also running.
That is the only scenario in which it is going to work because in that case Redis Sentinel 1
and Redis Sentinel 2 are going to agree, that Yes master is down and it is going to promote Slave
to Master Node.
Okay because Quorum is set to 2 and that's why
Redis Sentinel 1 and Redis Sentinel 2, both are available, then this process can work.
However in Production, usually we know that, you know, this is very unlikely that you are going to have
problem only with Redis and you are not going to have problem with Redis Sentinel. Whenever there is
any problem on that server.
Okay so that's the reason this is the setup which is not recommended.
Now let's discuss about another deployment.
So in this deployment we have 3 sentinels.
We have a Master and we have 2 Replicas
and here Quorum is set to 2 and this is a good approach.
If you have a minimum of 3 servers.
Okay.
So when you have minimum of 3 servers, all you can do is you can set up 1 Redis Sentinel on
each of the servers and one of the server can become Redis Master and remaining 2 servers are
going to be Slave 1 and Slave 2. So in this case whenever one node goes down as we have seen that, in
that case the other node is going to become Master Node.
So let's say M 1 was down and this is going to become a Master Node.
However let's say for some reason, you know this master is in a different subnet and Slave 1 and Slave 2
is in a different subnet. Let's say IP of this begins with something like 10. you know 0.0.*
and then these two Slave Nodes are from a different subnet. So it starts with, let's say something
from 10.4.0.*
and you know for some reason you have issue between this subnet, connecting to this subnet. Okay.
So in this case what is going to happen is, basically Redis Sentinel, you know, whole or this block or set
of users.
you know, are not going to be able to connect to Master Node. Even though you know, the actual service or on
the server is still up and running okay.
In fact Redis Sentinel 1 is also running and Redis is also running on this server.
However, since there is a network issue and network partition, between you know this subnet
and this subnet. So Redis Sentinel 2 and Redis Sentinel 3 is basically going to
agree that, Yes they are not able to reach Master. So they are going to convert this you know, one of the
Slaves to a Master and then the other remaining you know previous, the previous Master Node is going
to be turned into a Slave Node whenever that is available and going to join this network.
So in this scenario, basically lets say if we have a user which is User A. It is connected to you know, directly
to Master Node. So whenever it is performing all the right operation that is going to be you know,
So since this Master Node is still a Master Node. It is going to allow all the right operations but whenever
you know this issue is resolved and all of these networks are able to connect to each other.
This, since this master you know, state has been changed to a Slave Node by Sentinel.
So whenever it joints back the network. This M 1 is going to be a Slave of this New Master and whatever data
which has been written by User A, is going to get lost. Okay because this Slave will be trying to become
in the same state, which this New Master is.
However this issue does have a Mitigation.
So there is something called Minimum replicas to write configuration.
which you can set to 1,
In this case.
So whenever you have a scenario like this. In this case if User A is writing data to Master, Master will
not be having any Slaves. Okay.
It will not be having any Slaves to write or replicate its data to.
And since we have set minimum replicas to write is 1. So Master must have at least or minimum of 1 replicas,
where it needs to replicate its data to and if it is not able to find that one replica, it is not going
to allow any write operation. So user A, whenever it is trying to perform write operation, it is going
to get error.
And this is how User A, is going to get aware
Yes there is some issue, which is going on with the Redis Server.
However user A, still can successfully perform all the Read operations.
Now once you do this mitigation. There is again one more drawback is basically
that whenever both of the replicas are down. The master is not going to accept
writes.
Okay so let's say, when we have issue with S 1 and S 2 and both of them are not available. In that case master
server is not going to allow any write operation.
The reason being is because minimal replica is set to 1. So again whenever you are doing a set up in
your Production Environment, you can assess these scenarios that, Do you going to face any problem which
can occur because of Network Partition. Probably you can try to make all the Redis notes a part of the
same network partition.
Again
You know different kind of issues can produce a different behavior.
So always it is best, you do a good testing before deploying a Replication in a Production Environment.
Now let's discuss about another scenario. In this scenario we have only 2 Servers available for
Redis.
Okay so we have made one Redis Server to a Master Node and then we have converted one Redis Server
to a Slave Node.
Now in this scenario you see, we have not installed Redis sentinel on Redis Server.
So rather what is done here is
you know, there are set of 3 servers where application Server is or application is running, on
those nodes Redis Sentinels are installed.
okay and Redis
these Redis Sentinels are still going to monitor Master and Slave Nodes and going to work the same
way, which you know, which we have seen in this example.
Now the issue which can come here is, when you have you know, this application server is totally disconnected
with this Master and Slave Node. So let's say for some reason, when you know all of these is completely
disconnected with Master and Slave.
In that case you are going to have issues
and also for some reason. Lets say if there is no connectivity between this Applications Server and Master Server,
then you won't be able to set up basically this kind of setup.
Now let's discuss about another deployment scenario. In this scenario it is assumed that on the application
side also, you have only 2 servers available.
Okay.
So when I'm saying Application Side.
So basically this can be the servers where you have your, let's say Web Application running or any application
running, which is going to connect to the Redis Data Base. So in this case, we have only 2
servers available for Application and 2 servers are available for Redis Data Base.
So in this case we have installed 4 Sentinels.
Okay so we have 1 Sentinel which is installed on Application Server and then we have Redis Sentinel 2
which is also installed on Application Server.
Then we have Redis Sentinel 3, which is installed on the same server, where currently Redis master
is installed and then we have Redis Server or Redis Sentinel 4, which is installed on the same server where
Slave 1, is installed and in this case now since we have 4 Sentinels available.
So that's why, minimum number of Sentinel which needs to be agreed is set to 1.
oh, I'm sorry, which is set to 3
So you can see here Quorum is set to 3
and this is also a good approach.
If you have a minimum of, basically you have minimum of 3 servers, including your Client Server and Redis
Server. So client, by Client Server what I mean here is, the Application Server.
So somehow basically you need to achieve is, that you need to have minimum of 3 servers available
to install your Redis Sentinel, you know that is the precisely basically we are trying to say in all
of the scenarios. Okay in none of the scenarios, we have seen you know, perfectly fine working model
where we have less than 3 sentinels available.
So that's it about Redis High Availability and Sentinel Deployment. In Next Session we are
going to do a Hands-on Session. We are going to set up 3 servers and on those servers we are
going to set up a high availability using Sentinels.