Hi. I`m sending nginx logs to kafka. Each nginx goes to one exclusive partition. I guarantee this by asigning to each nginx a number (1 to 22), and I use that number for a custom partitoner's partition method as the key parameter: return Integer.parseInt(key) % partitions
I have 22 nginxs, so I have 22 partitions with data. The cluster has 5 brokers with ten partitions, so the total partitions for the topic is 50. In teory I could be able to start up to 22 consumers and each one should be able to consume from one partition. But what is happening is that I only can start three consumers: The first one consumes the ten partitions of the first broker. The second one consumes the ten partitions of the second broker. The third one consumes the remaining 2 partitions of the third broker. Any other consumer that I start in the same consumer group, doesn`t consumes anything, and the three consumers that are consuming are not balanced. I am using Kafka 0.7.1, with zookeeper and the java consumer connector, with the zk.connect property. What could it be happening? Thanks for the help! -- Javier Iseruk Architecture Team Phone: +5411-4640-8063
