Re: Kafka consumer configuration / performance issues

2016-10-05 Thread Shamik Banerjee
Sorry to bump this up, can anyone provide some input on this ? I need to make a call soon whether kafka is a good fit to our requirement. On Tuesday, October 4, 2016 8:57 PM, Shamik Banerjee wrote: Hi, I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation primarily is

Kafka consumer configuration / performance issues

2016-10-04 Thread Shamik Banerjee
Hi, I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation primarily is to improve performance where kafka would eliminate the constraint of pulling 10 messages at a time with a cap of 256kb. Here's a high-level scenario of my use case. I've a bunch of crawlers which are s