Hello and welcome to this lecture. In this lecture
we'll talk about ETCD in a high availability setup.
So this is really a prerequisite lecture for the next lecture where we talk about configuring Kubernetes
in a highly available mode.
Well one portion of that deals with configuring ETCD in HA mode.
So in this lecture we will discuss about ETCD in HA mode in the beginning of this course we took
a quick look at ETCD.
We will now recap real quick and more importantly focus on the cluster configuration on ETCD.
So let’s recap real quick and look at the number of nodes in the cluster, what RAFT protocol is etc.
So what is ETCD?
It's a distributed reliable key value store that is simple secure and fast.
So let's break it up.
Traditionally data was organized and stored in tables like this.
For example to store details about a number of individuals. A key value store stores information in the
form of documents or pages.
So each individual gets a document and all information about that individual is stored within that file.
These files can be in any format or structure and changes to one file does not affect the others.
In this case the working individuals can have their files with salary fields. While you could store and
retrieve simple key and values when your data gets complex.
you typically end up transacting in data formats like JSON or YAML.
So that’s what etcd is and how you quickly get started with it.
We also said that ETC is distributed.
So what does that mean.
And that is what we're going to focus in this lecture.
We had ETCD on a single server. But it’s a database and may be storing critical data.
So it is possible to have your datastore across multiple servers.
Now you have 3 servers, all running etcd, and all maintaining an identical copy of the database.
So if you lose one you still have two copies of your data.
Perfect.
But how does it ensure the data on all the nodes are consistent.
You can write to any instance and read your data from any instance. ETCD ensures that the same consistent
copy of the data is available on all instances at the same time.
So how does it do that? With reads,
its easy. Since the same data is available across all nodes,
you can easily read it from any nodes. But that is not the case with writes.
What if two write requests come in on two different instances? Which one goes through?
For example I have writes coming in for name set to john on one and with the name joe on the other.
Of course we cannot have two different data on two different nodes.
When I said ETCD can write through any instance, I wasn’t a 100% right.
ETCD does not process the writes on each node.
Instead, only one of the instances is responsible for processing the writes. Internally
the two nodes elect a leader among them. Of the total instances one node becomes the leader and the other
node becomes the followers.
If the writes came in through the leader node, then the leader processes the write. The leader makes sure
that the other nodes are sent a copy of the data.
If the writes come in through any of the other follower nodes, then they forward the writes to the leader
internally and then the leader processes the writes. Again
when the writes are processed, the leader ensures that copies of the write and distributed to other instances
in the cluster.
Thus a write is only considered complete, if the leader gets consent from other members in the cluster.
So how do they elect the leader among themselves? And how do they ensure a write is propogated across
all instances? ETCD implements distributed consensus using RAFT protocol.
Let's see how that works in a three node cluster.
When the cluster is setup we have 3 nodes that do not have a leader elected. RAFT algorithm uses
random timers for initiating requests.
For example a random timer is kicked off on the three managers the first one to finish the timer sends
out a request to the other node requesting permission to be the leader. The other managers on receiving
the request responds with their vote and the node assumes the Leader role. Now that it is elected the
leader it sends out notification at regular intervals to other masters informing them that it is continuing
to assume the role of the leader. In case the other nodes do not receive a notification from the leader
at some point in time which could either be due to the leader going down or losing network connectivity
the nodes initiate a re-election process among themselves and a new leader is identified going back
to our previous example where a right come seen it is processed by the leader and is replicated to other
nodes in the cluster the right is considered to be complete only once it is replicated to the other
instances in the cluster.
We said that the ETCD cluster is highly available. So even if we lose a node it should still function.
Say for example a new write comes in but one of the node is not responding. And hence the leader is only
able to write to 2 nodes in the cluster. Is the write considered to be complete? Does it wait for the
third node to be up? Or does it fail? A write is considered to be complete,
if it can be written on the majority of the nodes in the cluster. For example, in this case of the 3 nodes,
the majority is 2.
So if the data can be written on two of the nodes then the right is considered to be complete.
If the third node was to come on line then the data is copied to that as well.
So what is the majority.
Well a more appropriate term to use would be Quorum. Quorum is the minimum number of nodes that must
be available for the cluster to function properly or make a successful right in case of 3.
we know its 2. For any given number of nodes, the quorum is the total number of nodes divided by 2 + 1.
So Quorum of 3 nodes is 3/2 which is 1.5 + 1 equals 2.5.
If there is a .5 consider the whole number only so that's 2.
Similarly quorum of 5 nodes is 3.
So here is a table that shows the quorum of clusters of size 1 to 7. Quorum of 3 and 5 are what
we calculated. Quorum of 1 is 1 itself.
Meaning if you have a single non cluster none of these really apply.
If you lose that note everything's gone if you look at 2 and apply the same formula.
the quorum is 2 itself. 2/2 is 1 and 1 + 1 is 2.
So even if you have 2 instances in the cluster the majority is still 2. If one fails,
There is no quorum.
So rates won't be processed.
So having two instances is like having one instance it doesn't offer you any real value as quorum cannot
be met.
Which is why it is recommended to have a minimum of 3 instances in an ETCD cluster. That way
it offers a fault tolerance of at least 1 node.
If you lose one, you can still have quorum and the cluster will continue to function.
So the first column minus the second column gives you the fault tolerance. The number of nodes that you
can afford to lose while keeping the cluster alive.
So we have 1 to 7 nodes here. 1 and 2 are out of consideration.
So from 3 to 7 what do we consider?
As you can see 3 and 4 have the same fault tolerance of 1 and 5 and 6 have the same fault
tolerance of 2. When deciding on the number of master nodes,it is recommended to select an odd number
as highlighted in the table 3 or 5 or 7. Say we have a 6 node cluster.
So for example due to a disruption in the network it fails and causes the network to partition.
We have 4 nodes on one and 2 on the other.
In this case the group with 4 nodes have quorum and continues to operate normally. However if the
network got partitioned in a different way resulting in nodes being distributed equally between the two,
each group now has 3 nodes only.
But since we originally had 6 manager nodes, the quorum for the cluster to stay alive is 4
But if you look at the groups here neither of these groups have four managers to meet the quorum so
it results in a failed cluster.
So with even number of nodes there is possibility of the cluster failing during a network segmentation.
In case we had odd number of managers originally, say 7,
then after the network segmentation we have 4 on one segmented network and 3 on the other,
and so our cluster still lives on the group with 4 managers as it meets the quorum of 4.
No matter how the network segments there are better chances for your cluster to stay alive in case of
network segmentation with odd number of nodes.
So an odd number of nodes is preferred over even number having 5 is preferred over 6 and having
more than 5 nodes is really not necessary as 5 gives you enough for tolerance. To install ETCD on
a server.
download the latest supported binary. Extract it,
create the required directory structure. Copy over the certificate files generated for ETCD. We
discussed how to generate these certificates in detail in the TLS Certificates section. Then configure
the ETCD service.
What's important here is to note that the initial cluster option where we passing the peer's information
That’s how each etcd service knows that it is part of a cluster and where its peers are. Once installed
and configured use the etcdctl utility to store and retrieve data. ETCDCTL utility has
two API versions.
V2 and V3. So the commands work different in each version.
Version 2 is default.
However we will use version 3. So set an environment variable
ETCDCTL_API to 3, otherwise the below commands won’t work.
Run the etcdctl put command and specify the key as name and value as john. To retrieve data from
the etcdctl
get command with the key /name and it returns the value john.To get all keys run the etcdctl
get –keys-only
command. Going back to our design,
how many nodes, should our cluster have? In an HA environment
as you can see having 1 or 2 instances doesn’t really make any sense. As losing 1 node, in either
case will leave you without quorum and thus render the cluster not functional. Hence the minimum required
nodes in an HA setup is 3.
We also discussed why we prefer odd number of instances over even number. Having even number of instances
can leave the cluster without quorum in certain network partition scenarios.
So all that even number of nodes is out of scope. So we are left with 3 5 and 7 or any odd number above
that 3 is a good start but if you prefer a higher level of full tolerance then 5 is better. But anything
beyond that is just unnecessary.
So considering your environment the fault tolerance requirements and the cost that you can bear you
should be able to choose one number from this list.
In our case we go with 3.
So how does our design look now? With HA,
the minimum required number of nodes for fault tolerance is 3.
Now while it would be great to have 3 master nodes we are limited by our capacity of our laptop. So we
will just go with 2.
But if you're deploying the setup in another environment and have sufficient capacity feel free to go
with 3.
We also chose to go with the stacked topology where we will have the ETCD servers on the master nodes
itself.
Well that's it for this lecture.
Thank you for listening.
I will see you in the next.