1 00:00:00,570 --> 00:00:06,650 Hello and welcome to this lecture. We now look at high availability in kubernetes. 2 00:00:06,680 --> 00:00:11,830 So what happens when you lose the master node in your cluster? 3 00:00:12,130 --> 00:00:18,150 As long as the workers are up and containers are alive, your applications are still running. 4 00:00:18,160 --> 00:00:23,000 Users can access the application until things start to fail. 5 00:00:23,020 --> 00:00:26,420 For example a container or POD on the worker node crashes. 6 00:00:26,770 --> 00:00:32,710 Now, if that pod was part of a replicaset then the replication controller on the master needs to instruct 7 00:00:32,710 --> 00:00:35,150 the worker to load a new pod. 8 00:00:35,350 --> 00:00:40,590 But the Master is not available and so are the controllers and schedulers on the master. 9 00:00:40,600 --> 00:00:44,490 There is no one to recreate the pod and no one to schedule it on nodes. 10 00:00:44,530 --> 00:00:50,260 Similarly, since the kube-api server is not available you cannot access the cluster externally through 11 00:00:50,260 --> 00:00:58,030 kubectl tool or through API for management purposes. Which is why you must consider multiple master nodes in 12 00:00:58,030 --> 00:01:04,510 in a High availability configuration in your production environment. A high availability configuration is 13 00:01:04,510 --> 00:01:10,780 where you have redundancy across every component in the cluster so as to avoid a single point of failure. 14 00:01:11,650 --> 00:01:18,100 The master nodes, the worker nodes, the control plane components, the application of course, which we already 15 00:01:18,100 --> 00:01:21,820 have multiple copies in the form of replicasets and services. 16 00:01:21,880 --> 00:01:28,490 So our focus in this lecture is going to be on the master and control plane components. Let's take a 17 00:01:28,490 --> 00:01:30,390 better look at how it works. 18 00:01:30,440 --> 00:01:34,970 We’ve been looking at a 3 node cluster with 1 master and 2 worker nodes 19 00:01:34,970 --> 00:01:41,840 throughout this course. In this lecture, we will focus on just the master node. As we learned already the 20 00:01:41,840 --> 00:01:48,270 master node hosts the control plane components including the API, Controller Manager, Scheduler and ETCD 21 00:01:48,280 --> 00:01:55,640 server. In a HA setup, with an additional master node, you have the same components running on 22 00:01:55,640 --> 00:01:57,340 the new master as well. 23 00:01:57,530 --> 00:01:59,060 So how does that work. 24 00:01:59,090 --> 00:02:04,430 Running multiple instances of the same components? Are they going to do the same thing twice? 25 00:02:04,430 --> 00:02:07,100 How do they share the work among themselves? 26 00:02:07,160 --> 00:02:10,000 Well that differs based on what they do. 27 00:02:10,070 --> 00:02:16,310 We know that the API server is responsible for receiving requests and processing them or providing information 28 00:02:16,370 --> 00:02:20,650 about the cluster. They work on one request at a time. 29 00:02:20,660 --> 00:02:27,650 So the API servers on all cluster nodes can be alive and running at the same time in an active active 30 00:02:27,650 --> 00:02:30,460 mode. So far in this course 31 00:02:30,490 --> 00:02:35,220 we know that the kubectl utility talks to the API server to get things done 32 00:02:35,320 --> 00:02:42,160 and we point the kubectl utility to reach the master node at port 6443. 33 00:02:42,160 --> 00:02:47,220 That’s where the API server listens and this is configured in the kube-config file. 34 00:02:47,260 --> 00:02:51,740 Well now with 2 masters, where do we point the kubectl to? 35 00:02:52,180 --> 00:02:56,800 We can send a request to either one of them but we shouldn't be sending the same request to both of 36 00:02:56,800 --> 00:02:57,460 them. 37 00:02:57,460 --> 00:03:03,520 So it is better to have a load balancer of some kind configured in the front of the master nodes that 38 00:03:03,520 --> 00:03:07,030 split traffic between the API servers. 39 00:03:07,030 --> 00:03:11,710 So We then point the kubectl utility to that load balancer. 40 00:03:11,710 --> 00:03:18,550 You may use, NGINX or HA proxy or any other load balancer for this purpose. 41 00:03:18,620 --> 00:03:21,490 What about the scheduler and the controller manager. 42 00:03:21,560 --> 00:03:27,220 These are controllers that watch the State of the cluster and take actions. For example the controller 43 00:03:27,230 --> 00:03:32,600 manager consists of controllers like the replication controller that is constantly watching the state 44 00:03:32,600 --> 00:03:36,400 of PODs and taking necessary actions, like creating a new POD 45 00:03:36,410 --> 00:03:44,120 when one fails. If multiple instances of those run in parallel, then they might duplicate actions 46 00:03:44,120 --> 00:03:47,470 resulting in more parts than actually needed. 47 00:03:47,510 --> 00:03:49,700 The same is true with scheduler. 48 00:03:49,700 --> 00:03:53,180 As such they must not run in parallel. 49 00:03:53,180 --> 00:03:56,240 They run in an active standby mode. 50 00:03:56,240 --> 00:04:01,570 So then who decides which among the two is active and which is passive. 51 00:04:01,610 --> 00:04:05,210 This is achieved through a leader election process. 52 00:04:05,360 --> 00:04:06,770 So how does that work. 53 00:04:06,770 --> 00:04:11,880 Let's look at controller manager for instance. When a controller manager process is configured. 54 00:04:12,050 --> 00:04:16,940 You may specify the leader elect option which is by default set to true. 55 00:04:16,940 --> 00:04:23,390 With this option when the controller manager process starts it tries to gain a lease or a lock on an 56 00:04:23,520 --> 00:04:30,500 endpoint object in kubernetes named as kube-controller-manager endpoint. Which ever process first 57 00:04:30,620 --> 00:04:31,760 updates the endpoint. 58 00:04:31,760 --> 00:04:36,250 With this information gains the lease and becomes the active of the two. 59 00:04:36,470 --> 00:04:42,890 The other becomes passive it holds the lock for the lease duration specified using the leader-elect-lease- 60 00:04:42,890 --> 00:04:46,930 duration option which is by default set to 15 seconds. 61 00:04:47,180 --> 00:04:53,030 The active process then renews the lease every 10 seconds which is the default value for the option 62 00:04:53,420 --> 00:05:00,790 leader-elect-renew-deadline. Both the processes tries to become the leader every two seconds, set by 63 00:05:00,790 --> 00:05:03,460 the leader-elect-retry-period option. 64 00:05:03,460 --> 00:05:09,640 That way if one process fails maybe because the first must of crashes then the second process can acquire 65 00:05:09,640 --> 00:05:15,250 the lock and become the leader. The scheduler follows a similar approach and has the same command line 66 00:05:15,250 --> 00:05:19,280 options. Next up, ETCD. 67 00:05:19,350 --> 00:05:22,020 We discussed about ETCD earlier in this course. 68 00:05:22,380 --> 00:05:27,330 It's a good idea to go through that again now just to quickly refresh your memory as we're going to 69 00:05:27,330 --> 00:05:33,270 discuss some topics related to how ETCD works in this lecture. With ETCD, 70 00:05:33,270 --> 00:05:36,750 there are two topologies that you can configure in kubernetes. 71 00:05:36,960 --> 00:05:42,390 One is as it looks here and the same architecture we have been following through out this course, where 72 00:05:42,420 --> 00:05:49,800 ETCD is part of the kubernetes master nodes. It’s called as a stacked control plan nodes topology. 73 00:05:49,830 --> 00:05:56,440 This is easier to setup and easier to manage. And requires fewer nodes. But if one node goes down 74 00:05:56,500 --> 00:06:03,730 both an ETCD member and control plane instance is lost and redundancy is compromised. The other is 75 00:06:03,730 --> 00:06:11,950 where ETCD is separated from the control plane nodes and run on its own set of servers. This is a topology 76 00:06:11,980 --> 00:06:15,870 with external ETCD servers. Compared to the previous topology 77 00:06:15,880 --> 00:06:22,870 this is less risky as a failed control plane node does not impact the ETCD cluster and the data it stores. 78 00:06:23,110 --> 00:06:29,680 However it is harder to setup and requires twice the number of servers for the external etcd nodes. So 79 00:06:29,680 --> 00:06:35,590 remember, the API server is the only component that talks to the ETCD server and if you look into the 80 00:06:35,590 --> 00:06:42,070 API service configuration options, we have a set of options specifying where the ETCD server is. 81 00:06:42,100 --> 00:06:47,980 So regardless of the topology we use and wherever we configure ETCD servers, weather on the same server 82 00:06:48,010 --> 00:06:49,050 or on a separate server. 83 00:06:49,210 --> 00:06:56,810 ultimately we need to make sure that the API server is pointing to the right address of the ETCD servers. 84 00:06:56,820 --> 00:07:03,280 Now remember ETCD is a distributed system, so the API server or any other component that wishes to talk 85 00:07:03,280 --> 00:07:07,510 to it, can reach the ETCD server at any of its instances. 86 00:07:07,510 --> 00:07:12,210 You can read and write data through any of the available ETCD server instances. 87 00:07:12,220 --> 00:07:19,270 This is why we specify a list of etcd-servers in the kube-apiserver configuration. In the next lecture 88 00:07:19,270 --> 00:07:25,090 we discuss more about how ETCD server works in a cluster setup and the best practices around the number 89 00:07:25,090 --> 00:07:28,360 of recommended nodes in a cluster. 90 00:07:28,360 --> 00:07:32,830 So back to our design we had originally planned for a single master node in our cluster. 91 00:07:32,950 --> 00:07:38,450 Now with HA, we decided to configure multiple masters. 92 00:07:38,460 --> 00:07:44,160 We also mentioned about a load balancer for the API server. So we will have that as well. 93 00:07:44,160 --> 00:07:47,400 So we now have a total of five nodes in our cluster.