Partition is useful for increasing the degree of parallelism of consumers and to a certain degree, producers too. You can have multiple consumer groups consuming the same topic/partition.
Thanks, Jun On Sun, Jan 27, 2013 at 1:06 PM, S Ahmed <sahmed1...@gmail.com> wrote: > Say I create web application/service where customers signup, and they place > some javascript on their website which will then send over http a message > to my servers every time someone clicks on a link on their website. > > Each customer will send to their own custom subdomain like: > > customer1.example.com/api/put?linkId=1&...... > > Say I have 100,000 customers. > > 1. If all events are of the same type, what are the potential means I could > partition my topics? Or does it not make sense to? I'm confused as to > what I am reading, is a given kafka topic + paritition combination ONLY > allowed to be consumed by a single consumer group? If so, why is that? > the kafka server can only handle a single thread connecting to it?? > > 2. I will have a java servlet that will contain my producer (each front end > server will have the same servlet that will contain a producer). I want to > batch every x messages. From what I understand, my producer is something I > will create using a singleton correct? > > 3. I want my consumers to by dynamic in size, so during peak hours I want > to fire up more nodes to keep up with traffic, is there a production > worthy consumer daemon that I can use (or learn from) that is open sourced > somewhere? > > Much appreciated! >
