In this lecture, I'll answer the question where Kafka basically stores messages that are sent by producers
and consumed by consumers.
We have found out that you are able to start the multiple consumers and multiple producers and consumers
are able to read messages from the very first one in each topic.
In our case, we have read the messages from beginning in this cities that I have told you before that
Kafka stores all messages in the TMB Kafka dish logs folder by default, and that is the parameter to
the set in the settlement configuration of file.
Let me show you it quickly.
I'm going to be a Scholte open up Quantic folder and open up several properties file.
And here on this line you'll find this path where Kafka stores all messages.
Let's open up new terminal window and go to this folder.
So I'll do that here.
New window.
Let me get to the new desktop.
Like so and here, let's keep to don't forget last year and next we'll be off guard, this looks and
now let's list files here and look at that after creation of the new topic this week of this folder
and the Deleware, those files and this new folder.
But now there are a bunch of new folders with prefixes, consumer under a score of sets, and there
are 50 such kind of folders here.
Now, let me just say that the when we have started sending messages to see dystopic, Kafka has automatically
created a new topic to underscore consumer under a score of Seth.
And this topic has 50 partitions by default.
And you may notice that the Doors folder names start with consumer of Seth Zero.
And last one is consumer offsets.
Forty nine, and thus are simply 50 different folders for 50 partitions of new system.
Dobek Gold Consumer Offsets.
No, we are interested the contents of this zero folder record that this topic has just a single partition
and that's why you'll see only one folder with this name C zero.
Let's go into this folder.
Could see this zero and list files here and same as before we see here.
Only four files were that after creation of the new topic, we have examined contents of those files
and they were empty.
Let's have a look at those files now.
Let's explore contents of index file through its name and use card command based name.
And we see that this file is empty.
But what about this file with extension?
But look, let's go by its name, Gotthard base name.
And this file is not empty anymore.
This file is partially readable and you can distinguish just that.
We have sent to see this topic.
There was New York.
And if I go back to producer here, you will find this first message.
Last one is Toronto.
And if I go back here, you'll find Toronto here at the end of this file.
And that means that Kafka has thought all messages in this log file located in DMP.
It's less Kafka logs folder.
That's that symbol.
Of course, every message has additional sys information and we will get back to that later on in the
course.
But now you need to understand that Kafka simply stores old messages inside of the files.
Nothing else.
It works like a file system.
And that's why, thanks to those files, you are able anytime with messages from Kafka, Closter.
But certainly every message has its life time.
And after specific amount of time, Kafka will delete old messages in order to make room for new messages.
And consumers that cannot read messages in time will not be able to read them again.
But that's, of course, done for optimization.
Now, you should remember that all what Kafka does after it receives a message from producer, it adds
it to log file in specific Patison folder in our case folder name is zero.
OK, now let's go to the court and try to read log files of Kafka Breuker itself and see what was happening
there.
So let's go to a Scholte and go to Looks folder this one and forget that we have examined their Serrador
log file.
But you may notice that this file will be automatically cleaned up and the Kafka will store previous
logs in such files as, for example, this one, Selema would look dot and date.
And if there were enough logs, it archives them every single hour.
So please try to find here log file with the file name was dead goes this to your current date.
It should be at the end of this, at least in my case.
This file is here.
Let me open it up, close this site pane.
And because this file doesn't have extension to look Scholte, we will highlight every line with different
colors to make it happen.
Go to this plaintext and here in select language mode type look and select look from the dropdown and
no type of this file will set to look and we're able to read more easy.
And in this file you'll see a bunch of different messages.
Some of them are related to creation of system topic gold, consumer offsets.
I have told you about it before and you are able to find methods related to creation of this new system
topic.
For that, let's press comment F or edit and find and the here in the search bar, let's type fifty
end space and you should find a single message where fifty appears and this message is this one out.
The creation of topic to underscore consumer underscore offsets with fifty per.
And replication factor one, and this creation was successful, and that's what I have told you, Kafka
stalkers automatically created new system gold consumer offsets with 50 partitions.
OK, this is the first meaningful look here.
Also, if you scroll down, you'll find out that all partitions for this system topic were created at
Broecker with a zero.
And the reason for that is that we have just a single broker in our cluster.
And that's why all partitions for this topic were created there.
If there were more brokers available, then partitions will be spread among all available brokers.
We will try that later on in the growth and we will create our custom topics and you'll see that the
partitions will be spread among all available brokers.
OK.
Also, if you'll search for producers, let's search for producer.
You'll see some logs related to producers, but they basically don't tell us much.
Let's try to find messages related to consumers.
And for that, let's type here and consumer.
And you'll find messages like this one where a balance group council consumer eight nine eight zero
or stabilizing group calls or consumer is the same number.
Also, you'll find similar messages related to annuls or console consumer group with annuls or number
below.
You may find another COSO consumer group name with Syphax, in my case for one six six.
In your case, those numbers will be completely different.
Rajib that for consumers of messages we have launched to different consumers here, they are still running
and Kafka because automatically created new consumer groups with random numbers and does because every
consumer must belong to specific consumer group later on will try to start consumers with custom consumer
group names.
Now you'll see here random names like this one.
Also, let's try to find the name of the topic.
The name of the topic in our case is this and you won't find here any results.
And this is because looks are related to writing or reading from specific topic in our case, this very
low level and that they are not shown here.
Also, please search for what offsets like this and you'll find a bunch of messages related to offsets.
And offset is basically number of every message that is thought in Kafka topic.
Every message has unique of set number and the consumers are basically starting reading from specific
offset.
And when we have started this command with argument from beginning, this consumer has told Kafka that
it was three messages inside this topic from the very beginning, from the first one that has offset
number zero.
So I thought message in our case message, New York has offset zero and now it's a good time to end
this introduction of practice section where you were able to lounge producers, consumers, generate
some messages, consume them by consumers and less at the beginning of next section shortly.
Summarize what you have learned so far.
What is Kafka, Glouster, what is Breuker what a zookeeper.
And afterwards, we will talk in detail about such concepts as offset consumer group leader and so on.
So let's go on in the next section.
Bye bye.