Hi all, I am a software engineer from Zhihu.com. Kafka is so great and used heavily in Zhihu. There are probably over 2K Kafka brokers in total.
However, we are suffering from the problem that the performance degrades rapidly when the number of topics increases(sadly, we are using HDD). We are considering separating the logic layer and the storage layer of Kafka broker like Apache Pulsar. After the modification, a server may have several Kafka brokers and more topics. Those brokers all connect to a sole storage engine via RPC. The sole storage can do the load balancing work easily, and avoid creating too many files which hurts HDD. Is it hard? I think replacing the stuff in `Kafka.Log` would be enough, right? Regards, Lin.
