WEBVTT 00:04.420 --> 00:11.260 In this lecture, I'll answer the question where Kafka basically stores messages that are sent by producers 00:11.260 --> 00:13.330 and consumed by consumers. 00:13.660 --> 00:19.840 We have found out that you are able to start the multiple consumers and multiple producers and consumers 00:19.840 --> 00:24.230 are able to read messages from the very first one in each topic. 00:24.250 --> 00:30.530 In our case, we have read the messages from beginning in this cities that I have told you before that 00:30.640 --> 00:38.150 Kafka stores all messages in the TMB Kafka dish logs folder by default, and that is the parameter to 00:38.150 --> 00:40.780 the set in the settlement configuration of file. 00:41.080 --> 00:42.210 Let me show you it quickly. 00:42.210 --> 00:47.650 I'm going to be a Scholte open up Quantic folder and open up several properties file. 00:47.800 --> 00:53.250 And here on this line you'll find this path where Kafka stores all messages. 00:53.710 --> 00:56.830 Let's open up new terminal window and go to this folder. 00:57.040 --> 00:58.340 So I'll do that here. 00:58.690 --> 00:59.700 New window. 01:00.340 --> 01:02.020 Let me get to the new desktop. 01:02.920 --> 01:13.120 Like so and here, let's keep to don't forget last year and next we'll be off guard, this looks and 01:13.120 --> 01:20.200 now let's list files here and look at that after creation of the new topic this week of this folder 01:20.200 --> 01:23.760 and the Deleware, those files and this new folder. 01:24.160 --> 01:31.450 But now there are a bunch of new folders with prefixes, consumer under a score of sets, and there 01:31.450 --> 01:33.880 are 50 such kind of folders here. 01:34.390 --> 01:40.930 Now, let me just say that the when we have started sending messages to see dystopic, Kafka has automatically 01:40.930 --> 01:46.450 created a new topic to underscore consumer under a score of Seth. 01:46.900 --> 01:50.290 And this topic has 50 partitions by default. 01:50.740 --> 01:56.350 And you may notice that the Doors folder names start with consumer of Seth Zero. 01:56.560 --> 01:58.860 And last one is consumer offsets. 01:59.320 --> 02:06.490 Forty nine, and thus are simply 50 different folders for 50 partitions of new system. 02:06.490 --> 02:08.600 Dobek Gold Consumer Offsets. 02:09.070 --> 02:17.080 No, we are interested the contents of this zero folder record that this topic has just a single partition 02:17.080 --> 02:22.230 and that's why you'll see only one folder with this name C zero. 02:22.570 --> 02:24.040 Let's go into this folder. 02:24.130 --> 02:29.200 Could see this zero and list files here and same as before we see here. 02:29.200 --> 02:36.130 Only four files were that after creation of the new topic, we have examined contents of those files 02:36.130 --> 02:37.380 and they were empty. 02:37.720 --> 02:39.370 Let's have a look at those files now. 02:39.730 --> 02:45.640 Let's explore contents of index file through its name and use card command based name. 02:45.910 --> 02:47.910 And we see that this file is empty. 02:48.190 --> 02:50.390 But what about this file with extension? 02:50.410 --> 02:53.800 But look, let's go by its name, Gotthard base name. 02:53.800 --> 02:56.060 And this file is not empty anymore. 02:56.410 --> 03:00.460 This file is partially readable and you can distinguish just that. 03:00.460 --> 03:02.550 We have sent to see this topic. 03:02.920 --> 03:04.200 There was New York. 03:04.210 --> 03:08.080 And if I go back to producer here, you will find this first message. 03:08.380 --> 03:10.230 Last one is Toronto. 03:10.360 --> 03:14.590 And if I go back here, you'll find Toronto here at the end of this file. 03:14.950 --> 03:22.140 And that means that Kafka has thought all messages in this log file located in DMP. 03:22.150 --> 03:24.120 It's less Kafka logs folder. 03:24.340 --> 03:25.540 That's that symbol. 03:25.720 --> 03:30.430 Of course, every message has additional sys information and we will get back to that later on in the 03:30.430 --> 03:30.820 course. 03:31.120 --> 03:37.300 But now you need to understand that Kafka simply stores old messages inside of the files. 03:37.450 --> 03:38.350 Nothing else. 03:38.720 --> 03:40.570 It works like a file system. 03:40.900 --> 03:46.510 And that's why, thanks to those files, you are able anytime with messages from Kafka, Closter. 03:46.870 --> 03:49.360 But certainly every message has its life time. 03:49.360 --> 03:56.440 And after specific amount of time, Kafka will delete old messages in order to make room for new messages. 03:56.440 --> 04:00.700 And consumers that cannot read messages in time will not be able to read them again. 04:01.000 --> 04:03.040 But that's, of course, done for optimization. 04:03.430 --> 04:08.710 Now, you should remember that all what Kafka does after it receives a message from producer, it adds 04:08.710 --> 04:15.340 it to log file in specific Patison folder in our case folder name is zero. 04:15.690 --> 04:22.900 OK, now let's go to the court and try to read log files of Kafka Breuker itself and see what was happening 04:22.900 --> 04:23.250 there. 04:23.680 --> 04:31.140 So let's go to a Scholte and go to Looks folder this one and forget that we have examined their Serrador 04:31.150 --> 04:31.780 log file. 04:32.080 --> 04:40.030 But you may notice that this file will be automatically cleaned up and the Kafka will store previous 04:40.030 --> 04:46.390 logs in such files as, for example, this one, Selema would look dot and date. 04:46.840 --> 04:51.160 And if there were enough logs, it archives them every single hour. 04:51.400 --> 04:57.460 So please try to find here log file with the file name was dead goes this to your current date. 04:57.640 --> 05:00.250 It should be at the end of this, at least in my case. 05:00.250 --> 05:01.390 This file is here. 05:01.600 --> 05:05.050 Let me open it up, close this site pane. 05:05.350 --> 05:11.590 And because this file doesn't have extension to look Scholte, we will highlight every line with different 05:11.590 --> 05:13.450 colors to make it happen. 05:13.450 --> 05:21.190 Go to this plaintext and here in select language mode type look and select look from the dropdown and 05:21.190 --> 05:25.420 no type of this file will set to look and we're able to read more easy. 05:25.720 --> 05:29.410 And in this file you'll see a bunch of different messages. 05:29.620 --> 05:35.260 Some of them are related to creation of system topic gold, consumer offsets. 05:35.530 --> 05:42.040 I have told you about it before and you are able to find methods related to creation of this new system 05:42.040 --> 05:42.490 topic. 05:42.490 --> 05:49.870 For that, let's press comment F or edit and find and the here in the search bar, let's type fifty 05:49.870 --> 05:57.000 end space and you should find a single message where fifty appears and this message is this one out. 05:57.000 --> 06:02.200 The creation of topic to underscore consumer underscore offsets with fifty per. 06:02.790 --> 06:09.240 And replication factor one, and this creation was successful, and that's what I have told you, Kafka 06:09.480 --> 06:15.780 stalkers automatically created new system gold consumer offsets with 50 partitions. 06:16.230 --> 06:18.760 OK, this is the first meaningful look here. 06:19.140 --> 06:25.560 Also, if you scroll down, you'll find out that all partitions for this system topic were created at 06:25.560 --> 06:27.730 Broecker with a zero. 06:28.080 --> 06:32.130 And the reason for that is that we have just a single broker in our cluster. 06:32.190 --> 06:35.780 And that's why all partitions for this topic were created there. 06:36.270 --> 06:43.120 If there were more brokers available, then partitions will be spread among all available brokers. 06:43.260 --> 06:49.230 We will try that later on in the growth and we will create our custom topics and you'll see that the 06:49.230 --> 06:52.410 partitions will be spread among all available brokers. 06:52.620 --> 06:53.050 OK. 06:53.280 --> 06:57.690 Also, if you'll search for producers, let's search for producer. 06:58.350 --> 07:03.220 You'll see some logs related to producers, but they basically don't tell us much. 07:03.540 --> 07:06.510 Let's try to find messages related to consumers. 07:06.660 --> 07:09.440 And for that, let's type here and consumer. 07:09.720 --> 07:16.350 And you'll find messages like this one where a balance group council consumer eight nine eight zero 07:16.530 --> 07:20.270 or stabilizing group calls or consumer is the same number. 07:20.610 --> 07:26.730 Also, you'll find similar messages related to annuls or console consumer group with annuls or number 07:27.210 --> 07:27.660 below. 07:27.660 --> 07:33.240 You may find another COSO consumer group name with Syphax, in my case for one six six. 07:33.510 --> 07:36.720 In your case, those numbers will be completely different. 07:37.050 --> 07:43.260 Rajib that for consumers of messages we have launched to different consumers here, they are still running 07:43.650 --> 07:51.000 and Kafka because automatically created new consumer groups with random numbers and does because every 07:51.000 --> 07:58.890 consumer must belong to specific consumer group later on will try to start consumers with custom consumer 07:58.890 --> 07:59.580 group names. 07:59.820 --> 08:02.690 Now you'll see here random names like this one. 08:03.180 --> 08:06.060 Also, let's try to find the name of the topic. 08:06.610 --> 08:10.830 The name of the topic in our case is this and you won't find here any results. 08:11.070 --> 08:17.610 And this is because looks are related to writing or reading from specific topic in our case, this very 08:17.640 --> 08:20.580 low level and that they are not shown here. 08:21.090 --> 08:28.380 Also, please search for what offsets like this and you'll find a bunch of messages related to offsets. 08:28.740 --> 08:34.560 And offset is basically number of every message that is thought in Kafka topic. 08:34.740 --> 08:42.270 Every message has unique of set number and the consumers are basically starting reading from specific 08:42.270 --> 08:42.900 offset. 08:43.230 --> 08:49.980 And when we have started this command with argument from beginning, this consumer has told Kafka that 08:49.980 --> 08:55.230 it was three messages inside this topic from the very beginning, from the first one that has offset 08:55.230 --> 08:56.400 number zero. 08:56.520 --> 09:02.700 So I thought message in our case message, New York has offset zero and now it's a good time to end 09:02.700 --> 09:09.870 this introduction of practice section where you were able to lounge producers, consumers, generate 09:09.870 --> 09:15.480 some messages, consume them by consumers and less at the beginning of next section shortly. 09:15.480 --> 09:17.910 Summarize what you have learned so far. 09:18.150 --> 09:21.240 What is Kafka, Glouster, what is Breuker what a zookeeper. 09:21.450 --> 09:29.160 And afterwards, we will talk in detail about such concepts as offset consumer group leader and so on. 09:29.400 --> 09:31.250 So let's go on in the next section. 09:31.260 --> 09:31.800 Bye bye.