Re: Kafka consumer configuration / performance issues

2016-10-05 Thread Shamik Banerjee
Sorry to bump this up, can anyone provide some input on this ? I need to make a call soon whether kafka is a good fit to our requirement. On Tuesday, October 4, 2016 8:57 PM, Shamik Banerjee wrote: Hi, I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation prim

Kafka consumer configuration / performance issues

2016-10-04 Thread Shamik Banerjee
Hi, I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation primarily is to improve performance where kafka would eliminate the constraint of pulling 10 messages at a time with a cap of 256kb. Here's a high-level scenario of my use case. I've a bunch of crawlers which are s

Kafka consumer receiving same message multiple times

2016-09-27 Thread Shamik Banerjee
Hi, I've recently started using kafka to read documents coming through a web crawler. What I'm noticing is when I'm dealing with few million documents, the consumer is processing the same message over and over again. Looks like the data is not getting committed for some reason. This is not th