Could you check if the topic name is the same on all 5 brokers ? Can you send around the contents of the kafka log directory on all 5 brokers ?
Thanks, Neha On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk < javier.ise...@mercadolibre.com> wrote: > Hi. > I`m sending nginx logs to kafka. Each nginx goes to one exclusive > partition. > I guarantee this by asigning to each nginx a number (1 to 22), and I use > that number for a custom partitoner's partition method as the key > parameter: > return Integer.parseInt(key) % partitions > > I have 22 nginxs, so I have 22 partitions with data. > The cluster has 5 brokers with ten partitions, so the total partitions for > the topic is 50. > > In teory I could be able to start up to 22 consumers and each one should be > able to consume from one partition. > But what is happening is that I only can start three consumers: > The first one consumes the ten partitions of the first broker. > The second one consumes the ten partitions of the second broker. > The third one consumes the remaining 2 partitions of the third broker. > > Any other consumer that I start in the same consumer group, doesn`t > consumes anything, and the three consumers that are consuming are not > balanced. > > I am using Kafka 0.7.1, with zookeeper and the java consumer connector, > with the zk.connect property. > > What could it be happening? > > Thanks for the help! > > -- > Javier Iseruk > Architecture Team > Phone: +5411-4640-8063 >
