WEBVTT 00:04.210 --> 00:08.980 OK, now, you know, structure of the message to distort and Kafka brokers, and now is a good time 00:08.980 --> 00:15.060 to go on and talk about how topics are actually stored on brokers and what is partition. 00:15.490 --> 00:21.570 So every topic may exist on different brokers that are included in a Kafka class. 00:21.760 --> 00:27.700 For example, here on this diagram, you see that topic A is present at Broker Zero and and broker one. 00:27.970 --> 00:33.090 And Topic B is present on Broker one, broker and broker at three. 00:33.640 --> 00:40.120 Another topic, topic C is present only at broker to notice that on this diagram I have removed the 00:40.120 --> 00:41.410 zookeeper for simplicity. 00:41.530 --> 00:46.110 But it is always there without zookeeper, Kafka broker is not able to operate. 00:46.480 --> 00:47.830 So you might ask me a question. 00:47.830 --> 00:50.640 Why do we need the same topic on different brokers? 00:51.370 --> 00:54.220 Why don't just to create a single topic? 00:54.220 --> 00:58.510 A single broker answer is pretty simple for fault tolerance. 00:58.810 --> 01:05.290 If in such case Broker two will fail, old messages in Topic C will be lost and no one will be able 01:05.290 --> 01:11.890 to produce new messages to Topic C and consumer messages from this topic because broker that was serving 01:12.040 --> 01:15.210 old messages inside of this topic, topic C is gone. 01:16.210 --> 01:21.970 In this case, when we talk about topic A and if, for example, a broker or zero fails, broker one 01:21.970 --> 01:28.720 will be able to still save you messages that will arrive to topic EHP and it will able to sell reasonable 01:28.720 --> 01:32.740 requests from consumers that will ask for data from this topic. 01:32.740 --> 01:35.020 A but how? 01:35.020 --> 01:41.650 Actually messages are spread among different brokers in such case, when same topic is present on different 01:41.650 --> 01:42.220 brokers. 01:42.520 --> 01:44.980 For that, Kafka uses partitions. 01:44.980 --> 01:51.580 And here in this example, you see that topic A has two partitions partition and partition one one partition 01:51.580 --> 01:54.870 is located and broker zero and second one is located. 01:54.910 --> 01:55.600 Broker one. 01:56.230 --> 02:02.110 If we will talk about topic B, it has in this example three partitions starting from partition number 02:02.110 --> 02:02.580 zero. 02:02.800 --> 02:08.230 So this partition is located and broker number one, this partition number one is located at another 02:08.230 --> 02:12.670 broker, broker two and Partition two is located at the last broker number three. 02:13.510 --> 02:20.110 Topic C has this example, only a single partition partition zero, and it is located only on a single 02:20.110 --> 02:20.700 broker. 02:20.950 --> 02:26.680 Also this diagram you see one more topic topic deal that has to partition partition zero and partition 02:26.680 --> 02:30.820 one that allocated both on one broker, a broker or zero. 02:30.970 --> 02:37.330 It is also possible also you are able to create the topics with hundreds of different partitions that 02:37.330 --> 02:39.760 will be spread among multiple brokers. 02:40.180 --> 02:43.360 But what is an idea behind creation of those partitions? 02:43.570 --> 02:49.390 Why don't just create the topic was just single partition as shown here in this example? 02:49.930 --> 02:56.020 Because, as you know, every message is read into the file and file is stored on a hard drive of every 02:56.020 --> 02:56.620 computer. 02:56.950 --> 03:03.610 And if there will be a lot of messages that are produced simultaneously by many producers, single computer 03:03.610 --> 03:08.770 will be able simply to write the such amount of data weekly to the hard drive. 03:09.190 --> 03:15.130 This way, if there will be multiple partitions spread among different computers, they will perform 03:15.130 --> 03:18.530 right operations of matter just much, much more quicker. 03:18.530 --> 03:20.830 The same relates to read operations. 03:21.130 --> 03:26.890 There are many consumers and if, for example, one thousand consumers will try to read data from Topic 03:26.890 --> 03:31.840 C Partitions zero, this program will simply go down due to lack of resources. 03:32.260 --> 03:38.980 But if this topic will have, for example, hundreds of partitions spread among different brokers, 03:39.370 --> 03:44.040 this job of supplying data to consumers will be much, much easier. 03:44.320 --> 03:50.860 That is the reason for creation of different positions that are created basically on different brokers. 03:51.310 --> 03:53.980 Also, it makes topic for tolerant. 03:53.980 --> 04:01.090 And in this example, you, for example, broker a zero will fail topic A will still be present on Broecker 04:01.090 --> 04:07.930 one and it will still accept new messages and accept the new read request from consumers. 04:08.650 --> 04:10.990 That's why partitions are needed. 04:11.530 --> 04:14.170 Also in this diagram, I have shown you this topic. 04:14.500 --> 04:21.490 Barbaras in order to make you understand that multiple partitions in the same topic may exist on a same 04:21.490 --> 04:21.910 block. 04:22.000 --> 04:22.780 It is possible. 04:23.440 --> 04:24.250 OK, let's proceed. 04:24.250 --> 04:30.670 And next, let me explain you how messages are spread among different partitions when they are produced 04:30.670 --> 04:31.690 by producers. 04:31.960 --> 04:32.770 So see you next. 04:32.790 --> 04:33.100 My.