I'm not sure if this relates directly to your problem, but you are using
non-standard topic names because of the @ character. Topic names should
only include alphanumeric plus hyphen and underscore. Have you checked your
logs for any errors because of processing these topic names?
Regards,
Dennis
Hi again!
I have more info to share.
What I'm seeing in zookeeper is that the consumers are owning almost the
same partitions quantity, no matter if that partition has data or not. So
for example:
Consumer 1: 17 partitions (all with data)
Consumer 2: 17 partitions (only five with data)
Consumer 3:
Thanks for the answer. I send you the info:
I am ommiting all the files in each directory, just leaves one for showing
the structure.
There are two topics: access_log@nginx-nginx-api-master and
access_log@nginx-nginx-api-v1.
The consumer uses a whitelist with the regex access_log@nginx-nginx-api-.*
Could you check if the topic name is the same on all 5 brokers ? Can you
send around the contents of the kafka log directory on all 5 brokers ?
Thanks,
Neha
On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk <
javier.ise...@mercadolibre.com> wrote:
> Hi.
> I`m sending nginx logs to kafka. Each nginx
Hi.
I`m sending nginx logs to kafka. Each nginx goes to one exclusive partition.
I guarantee this by asigning to each nginx a number (1 to 22), and I use
that number for a custom partitoner's partition method as the key parameter:
return Integer.parseInt(key) % partitions
I have 22 nginxs, so I
