Thanks for the answer. I send you the info: I am ommiting all the files in each directory, just leaves one for showing the structure. There are two topics: access_log@nginx-nginx-api-master and access_log@nginx-nginx-api-v1. The consumer uses a whitelist with the regex access_log@nginx-nginx-api-.*, so both topis are consumed I don't see any problem with the topics names.
Broker 0: /kafka/logs> tree . |-- access_log@nginx-nginx-api-master-0 | `-- 00000000104220676901.kafka |-- access_log@nginx-nginx-api-master-1 | `-- 00000001848656795194.kafka |-- access_log@nginx-nginx-api-master-2 | `-- 00000000205422337816.kafka |-- access_log@nginx-nginx-api-master-3 | `-- 00000000065714271675.kafka |-- access_log@nginx-nginx-api-master-4 | `-- 00000002042994150834.kafka |-- access_log@nginx-nginx-api-master-5 | `-- 00000003554704806448.kafka |-- access_log@nginx-nginx-api-master-6 | `-- 00000005445803344687.kafka |-- access_log@nginx-nginx-api-master-7 | `-- 00000003606342657110.kafka |-- access_log@nginx-nginx-api-master-8 | `-- 00000002693751681000.kafka |-- access_log@nginx-nginx-api-master-9 | `-- 00000001393405947605.kafka |-- access_log@nginx-nginx-api-v1-0 | `-- 00000000132598986468.kafka |-- access_log@nginx-nginx-api-v1-1 | `-- 00000000137269970386.kafka |-- access_log@nginx-nginx-api-v1-2 | `-- 00000000136972308161.kafka |-- access_log@nginx-nginx-api-v1-3 | `-- 00000000146486674802.kafka |-- access_log@nginx-nginx-api-v1-6 | `-- 00000000009125337012.kafka |-- access_log@nginx-nginx-api-v1-8 | `-- 00000000009010149647.kafka broker 1: /kafka/logs> tree . |-- access_log@nginx-nginx-api-master-0 | `-- 00000002313153902757.kafka |-- access_log@nginx-nginx-api-master-1 | `-- 00000003097259627500.kafka |-- access_log@nginx-nginx-api-master-2 | `-- 00000002959536779137.kafka |-- access_log@nginx-nginx-api-master-3 | `-- 00000003053863102518.kafka |-- access_log@nginx-nginx-api-master-4 | `-- 00000002187535473763.kafka |-- access_log@nginx-nginx-api-master-5 | `-- 00000004427181341253.kafka |-- access_log@nginx-nginx-api-master-6 | `-- 00000002356845031168.kafka |-- access_log@nginx-nginx-api-master-7 | `-- 00000002286128315593.kafka |-- access_log@nginx-nginx-api-master-8 | `-- 00000002240322631141.kafka |-- access_log@nginx-nginx-api-master-9 | `-- 00000002298179990598.kafka |-- access_log@nginx-nginx-api-v1-0 | `-- 00000000000111649340.kafka |-- access_log@nginx-nginx-api-v1-1 | `-- 00000000000112654964.kafka |-- access_log@nginx-nginx-api-v1-2 | `-- 00000000000110500958.kafka |-- access_log@nginx-nginx-api-v1-3 | `-- 00000000000107970659.kafka broker 2: /kafka/logs> tree . |-- access_log@nginx-nginx-api-master-0 | `-- 00000001295587955823.kafka |-- access_log@nginx-nginx-api-master-1 | `-- 00000001280489056047.kafka |-- access_log@nginx-nginx-api-master-2 | `-- 00000000465978408295.kafka |-- access_log@nginx-nginx-api-master-3 | `-- 00000000459780591967.kafka |-- access_log@nginx-nginx-api-master-4 | `-- 00000000558193144600.kafka |-- access_log@nginx-nginx-api-master-5 | `-- 00000000556797739267.kafka |-- access_log@nginx-nginx-api-master-6 | `-- 00000000553188940491.kafka |-- access_log@nginx-nginx-api-master-7 | `-- 00000000562912390935.kafka |-- access_log@nginx-nginx-api-master-8 | `-- 00000000594913040282.kafka |-- access_log@nginx-nginx-api-master-9 | `-- 00000000691253299404.kafka |-- access_log@nginx-nginx-api-v1-0 | `-- 00000000005593211143.kafka |-- access_log@nginx-nginx-api-v1-1 | `-- 00000000005569669640.kafka |-- access_log@nginx-nginx-api-v1-2 | `-- 00000000005572829089.kafka |-- access_log@nginx-nginx-api-v1-3 | `-- 00000000005598264785.kafka broker 3: /kafka/logs> tree . |-- access_log@nginx-nginx-api-master-0 | `-- 00000000000048203157.kafka |-- access_log@nginx-nginx-api-master-1 | `-- 00000000000003173076.kaf|-- access_log@nginx-nginx-api-v1-0 | `-- 00000000000000223076.kafka |-- access_log@nginx-nginx-api-v1-2 | `-- 00000000000000135488.kafka broker 4: /kafka/logs> tree . |-- access_log@nginx-nginx-api-master-0 | `-- 00000000000101790264.kafka |-- access_log@nginx-nginx-api-master-1 | `-- 00000000000650193547.kafka |-- access_log@nginx-nginx-api-master-4 | `-- 00000000000516227252.kafka |-- access_log@nginx-nginx-api-master-5 | `-- 00000000000062572953.kafka |-- access_log@nginx-nginx-api-master-6 | `-- 00000000000566421014.kafka |-- access_log@nginx-nginx-api-master-8 | `-- 00000000000892503077.kafka |-- access_log@nginx-nginx-api-v1-0 | `-- 00000000000011969830.kafka |-- access_log@nginx-nginx-api-v1-2 | `-- 00000000000019642155.kafka |-- access_log@nginx-nginx-api-v1-3 | `-- 00000000000000636597.kafka Thanks! Javier Iseruk On Thu, May 9, 2013 at 1:18 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Could you check if the topic name is the same on all 5 brokers ? Can you > send around the contents of the kafka log directory on all 5 brokers ? > > Thanks, > Neha > > > On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk < > javier.ise...@mercadolibre.com> wrote: > > > Hi. > > I`m sending nginx logs to kafka. Each nginx goes to one exclusive > > partition. > > I guarantee this by asigning to each nginx a number (1 to 22), and I use > > that number for a custom partitoner's partition method as the key > > parameter: > > return Integer.parseInt(key) % partitions > > > > I have 22 nginxs, so I have 22 partitions with data. > > The cluster has 5 brokers with ten partitions, so the total partitions > for > > the topic is 50. > > > > In teory I could be able to start up to 22 consumers and each one should > be > > able to consume from one partition. > > But what is happening is that I only can start three consumers: > > The first one consumes the ten partitions of the first broker. > > The second one consumes the ten partitions of the second broker. > > The third one consumes the remaining 2 partitions of the third broker. > > > > Any other consumer that I start in the same consumer group, doesn`t > > consumes anything, and the three consumers that are consuming are not > > balanced. > > > > I am using Kafka 0.7.1, with zookeeper and the java consumer connector, > > with the zk.connect property. > > > > What could it be happening? > > > > Thanks for the help! > > > > -- > > Javier Iseruk > > Architecture Team > > Phone: +5411-4640-8063 > > > -- Javier Iseruk Architecture Team Phone: +5411-4640-8063
