Multiple topics strategy

2016-07-10 Thread Vladi Feigin
Hi All, Will appreciate if you share your thoughts based on your experience regarding the following question: What's a recommended strategy for Kafka multiple topics? We have just a few topics and hundreds of event types and a lot of producers Having just a few topics causes our consumers (many of

Kafka on AWs EMR

2016-07-10 Thread Snehalata Nagaje
Hi All, I wanted to setup kafka on AWS EMR? But I could not find any documentation. Can you please help me on this? Thanks, Snehalata

Re: Multiple topics strategy

2016-07-10 Thread rick
Without knowing much else about your requirements - Yes, your consumers should not be listening to topics they are not interested in. There's nothing wrong with having more topics. > On Jul 10, 2016, at 10:37 AM, Vladi Feigin wrote: > > Hi All, > > Will appreciate if you share your thoughts

Re: Kafka on AWs EMR

2016-07-10 Thread rick
Kafka is in no way related to map reduce, this doesn't really make any sense. You could have consumers running in Hadoop listening to Kafka topics - see the Samza project. > On Jul 10, 2016, at 11:16 AM, Snehalata Nagaje > wrote: > > > > Hi All, > > > I wanted to setup kafka on AWS EMR?

Re: Kafka on AWs EMR

2016-07-10 Thread Snehalata Nagaje
Though it has nothing to do map reduce, I was referingto EMR machines vs EC2 instance on amaozone? We an install other application on aws emr apart from what AWS provides, isn't it right? - Original Message - From: r...@chartbeat.com To: users@kafka.apache.org Sent: Sunday, July 10,

Re: Multiple topics strategy

2016-07-10 Thread Vladi Feigin
thank you for the answer Is there any kind of limit on a number of topics in Kafka? If one consumer (can not do more than one consumer in a group) reads lets say 100 topics it can be an issue, isn't it? On Sun, Jul 10, 2016 at 6:53 PM, wrote: > Without knowing much else about your requirement

Re: Kafka - offset preservation

2016-07-10 Thread R Krishna
Which version is this? I am a new user myself, are you using the same consumer group name? In V0.9, once connected, broker stores consumer offsets per consumer group, and continues from where it left off, the earliest/latest are for the first time only. On Wed, Jul 6, 2016 at 6:19 AM, Pawel Huszcz