Hi,
ZK chroot allows the Zookeeper ensemble to be shared with other
applications, including other Kafka clusters, without a conflict.
Kafka brokers from different clusters can connect to a single ZK
ensemble consists from 5 nodes (metadata), I will explain below why I
have chosen 5 nodes.
One can have many brokers which can be failed/replaced/removed/added
at any time (data).
These 5 nodes of ZK ensemble are distributed against multiple
availability zones (3) makes metadata to be more highly available
compared to having nodes in 2 AZs.
AZ-A AZ-B AZ-C
+-----------+ +------------+ +-----------+
|+---+ +---+| | +---+ | |+---+ +---+|
||ZK1| |ZK2|| | |ZK3| | ||ZK4| |ZK5||
|+---+ +---+| | +---+ | |+---+ +---+|
+-----------+ +------------+ +-----------+
An enterprise may have two datacenter at their premises, for third
availability zone one can have it as a cloud service privately
connected (VPN) to enterprise network.
This way metadata and data would have different level of availability
and it enable enterprises to design a more cost effective solution by
separating metadata and data service layer
Quoting from Jay Kreps. -"you might have tens or hundreds of data
nodes but metadata remains tiny"
https://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepse=
n#:~:text=3Dyou%20might%20have%20tens%20or%20hundreds%20of%20data%20nodes%2=
C%20but%20metadata%20remains%20tiny
The question is, if Kraft protocol is ever able to have this functionality?
- chroot may implemented as different topics for the
"__cluster_metadata" (__cluster_metadata_clusterID)
- brokers that are from different clusters can cache metadata
according the clusterID
- controllers (quorum voters) does not care which brokers (forming a
cluster) that they are serving, data nodes come and go, only the
relevant metadata is sent for each cluster.
Should I invest time for this, maybe creating a KIP?
Thank you. Kind regards,
Omer Siar Baysal
