WEBVTT 00:04.330 --> 00:10.960 So far, we have discussed a lot and you understand that every topic may contain multiple partitions 00:10.960 --> 00:15.410 and those partitions may be somehow spread between different brokers. 00:15.820 --> 00:22.660 Also, if you stop replication of fact or on the topic level, you could create replicas of every message 00:22.660 --> 00:25.060 in every partition of specific topic. 00:25.240 --> 00:31.630 And for that, the leader must be elected and this leader will be responsible for read and write operations 00:31.630 --> 00:32.830 for this partition. 00:33.010 --> 00:39.310 And also it will be responsible for replication of old messages from this partition to all followers, 00:39.850 --> 00:40.930 but no questions. 00:41.380 --> 00:46.520 Who decides which broker will be leader for a particular partition? 00:46.900 --> 00:51.820 So why, in this case, a broker one was chosen as leader for partition zero. 00:51.910 --> 00:55.010 And why broker a zero and broker two for us? 00:55.570 --> 00:56.770 Same question here. 00:57.070 --> 01:03.850 How it was decided that the Partition Zero will exist on broker a zero, partition one, on broker one 01:03.850 --> 01:04.540 and so on. 01:05.110 --> 01:10.840 Why not just to create the partition zero, partition one and partition two on the same, broker a zero. 01:11.110 --> 01:13.930 Why they were spread and who decides that? 01:14.560 --> 01:15.580 Another question. 01:15.880 --> 01:23.590 What if, for example, in this case, broker one fails like this, what will happen with this partition? 01:24.160 --> 01:28.700 This partition must be somehow reassigned to another broker. 01:28.870 --> 01:35.890 So this partition must be taken like that and created, for example, at Broker Zero when this partition 01:35.890 --> 01:43.210 fails on this broker like this, because this topic, topic A in our example, must have three partitions 01:43.480 --> 01:47.080 and all of them must exist somewhere on brokers. 01:47.410 --> 01:55.360 And if this broker has failed and only two brokers remained, we need to decide where to move this partition, 01:55.360 --> 01:59.710 either to this broker or to another broker like that. 02:00.900 --> 02:07.020 Also, we need to decide whether we are able to keep messages that were created in this partition or 02:07.020 --> 02:13.890 not, if there was an application set up for this partition partition one and all those messages were 02:13.890 --> 02:20.700 replicated, for example, to broker a zero, then we are able to retain those logs here after failure 02:20.700 --> 02:21.510 or broker one. 02:21.960 --> 02:28.410 But if there was no replication here, then partition one will be created, edbrooke or zero. 02:28.620 --> 02:36.150 But all messages that existed there before on Broecker, one will be simply lost again in case if there 02:36.150 --> 02:37.260 is no replication. 02:37.740 --> 02:44.460 But if new messages arrive to partition one, they will be successfully written to that partition in 02:44.460 --> 02:46.530 our case by Broecker zero. 02:46.890 --> 02:48.780 If again ROKR one has failed. 02:49.290 --> 02:57.420 And again, somehow someone needs to decide where this partition one needs to be recreated after failure 02:57.420 --> 02:58.580 of specific Breuker. 02:59.010 --> 03:05.850 But what if Broecker zero fails, then partition zero and partition one must be reassigned to remaining 03:05.850 --> 03:06.330 brokers. 03:06.330 --> 03:14.040 In our case, if only one broker remains, then it will halt all partitions from zero till second partition. 03:14.850 --> 03:23.010 And those administrative tasks require a lot of actually work and somehow someone needs to do all of 03:23.010 --> 03:23.340 that. 03:23.550 --> 03:28.900 Elect leaders, reassign partitions, create new partitions for new topics and so on. 03:29.730 --> 03:33.510 You may think that it is a job of zookeeper, but it is not the case. 03:33.720 --> 03:40.350 If a job of one of Kafka brokers and such a broker is called controller. 03:40.800 --> 03:46.970 And on this diagram you see that one of the brokers was selected as controller. 03:47.220 --> 03:53.340 It is highlighted here in purple and this controller is basically elected by the zookeeper. 03:53.430 --> 03:56.730 That is zookeeper's job, just elect controller. 03:56.880 --> 04:02.700 And now it's control the responsibility to create different partitions, to assign them to different 04:02.700 --> 04:06.240 brokers, reassign in case of failure of brokers. 04:06.540 --> 04:14.310 Then if there was a duplication configured on topic level, then that is job to select leader for specific 04:14.310 --> 04:16.740 position and select forewords. 04:16.980 --> 04:22.140 If there are files, then the controller decides who will become leader instead. 04:22.860 --> 04:29.580 So a controller is just one of the brokers in Kafka Latham that is elected automatically by Zookeeper 04:29.940 --> 04:34.800 and the controller is responsible for such operations as I have just described you. 04:34.980 --> 04:38.490 So reassign in competitions, election of leader and so on. 04:38.940 --> 04:45.800 If there are many, many topics with multiple partitions and are replicas set up for every topic, then 04:46.080 --> 04:49.410 overhead for this controller is a really, really high. 04:49.620 --> 04:55.140 And of course, you need to count on that and you need to plan resources all for your Kafka brockers 04:55.230 --> 04:56.040 accordingly. 04:56.670 --> 05:01.830 If controller or fails, then new controller is elected again by zookeeper. 05:02.700 --> 05:05.760 That's what is controller and that's what controller does. 05:06.300 --> 05:07.200 OK, let's proceed. 05:07.200 --> 05:12.990 And next to let me talk about how producers produce messages to different partitions and how those messages 05:12.990 --> 05:14.220 are written by Kafka. 05:14.220 --> 05:15.960 Brockers see you next by.