We're setting up a new kafka cluster with many brokers. How can we set a topic partition count to be less than total # of brokers in the cluster. The reason for this is we figure our cluster will have more sequential IO and and thus significantly higher performance than a setup where there's 1 partition per broker and a dozen+ topics.
I asked this question to someone on #kafka IRC, and they mentioned we could use the Producer Partitioner class to accomplish this. Has anyone run this in production, what is the complexity involved in this approach long term, and/or any other approaches we could take besides splitting up our single kafka cluster into many smaller ones. Thanks, Jason
