So far, we have discussed a lot and you understand that every topic may contain multiple partitions
and those partitions may be somehow spread between different brokers.
Also, if you stop replication of fact or on the topic level, you could create replicas of every message
in every partition of specific topic.
And for that, the leader must be elected and this leader will be responsible for read and write operations
for this partition.
And also it will be responsible for replication of old messages from this partition to all followers,
but no questions.
Who decides which broker will be leader for a particular partition?
So why, in this case, a broker one was chosen as leader for partition zero.
And why broker a zero and broker two for us?
Same question here.
How it was decided that the Partition Zero will exist on broker a zero, partition one, on broker one
and so on.
Why not just to create the partition zero, partition one and partition two on the same, broker a zero.
Why they were spread and who decides that?
Another question.
What if, for example, in this case, broker one fails like this, what will happen with this partition?
This partition must be somehow reassigned to another broker.
So this partition must be taken like that and created, for example, at Broker Zero when this partition
fails on this broker like this, because this topic, topic A in our example, must have three partitions
and all of them must exist somewhere on brokers.
And if this broker has failed and only two brokers remained, we need to decide where to move this partition,
either to this broker or to another broker like that.
Also, we need to decide whether we are able to keep messages that were created in this partition or
not, if there was an application set up for this partition partition one and all those messages were
replicated, for example, to broker a zero, then we are able to retain those logs here after failure
or broker one.
But if there was no replication here, then partition one will be created, edbrooke or zero.
But all messages that existed there before on Broecker, one will be simply lost again in case if there
is no replication.
But if new messages arrive to partition one, they will be successfully written to that partition in
our case by Broecker zero.
If again ROKR one has failed.
And again, somehow someone needs to decide where this partition one needs to be recreated after failure
of specific Breuker.
But what if Broecker zero fails, then partition zero and partition one must be reassigned to remaining
brokers.
In our case, if only one broker remains, then it will halt all partitions from zero till second partition.
And those administrative tasks require a lot of actually work and somehow someone needs to do all of
that.
Elect leaders, reassign partitions, create new partitions for new topics and so on.
You may think that it is a job of zookeeper, but it is not the case.
If a job of one of Kafka brokers and such a broker is called controller.
And on this diagram you see that one of the brokers was selected as controller.
It is highlighted here in purple and this controller is basically elected by the zookeeper.
That is zookeeper's job, just elect controller.
And now it's control the responsibility to create different partitions, to assign them to different
brokers, reassign in case of failure of brokers.
Then if there was a duplication configured on topic level, then that is job to select leader for specific
position and select forewords.
If there are files, then the controller decides who will become leader instead.
So a controller is just one of the brokers in Kafka Latham that is elected automatically by Zookeeper
and the controller is responsible for such operations as I have just described you.
So reassign in competitions, election of leader and so on.
If there are many, many topics with multiple partitions and are replicas set up for every topic, then
overhead for this controller is a really, really high.
And of course, you need to count on that and you need to plan resources all for your Kafka brockers
accordingly.
If controller or fails, then new controller is elected again by zookeeper.
That's what is controller and that's what controller does.
OK, let's proceed.
And next to let me talk about how producers produce messages to different partitions and how those messages
are written by Kafka.
Brockers see you next by.