Hi, I think I'm a bit confused. When you say "one producer per topic", do you mean one instance of the JVM application that's producing per topic?
Thanks Tom On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif <hafsa.a...@matchinguu.com> wrote: > Tom, > > Thank you for your answer. No, I am talking about one PRODUCER for each > topic, not one instance of same producer class. I am asking for general > concept only. > Actually we are just growing and not so much far from the case of 1 > million records per sec. Just considering our future case, I need your > suggestion in more detail, that in general is it a good practice to: > 1. Prepare a single producer for multiple topics (consider 10 topics) . > 2. Prepare 10 producers for 10 topics respectively. > > Your answer is quite satisfying for me, but I need more details so that I > can convince my team in a good way. > > Best Regards, > Hafsa > > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>: > > > Is that "one instance of the producer class per topic"? I'd recommend > just > > having a single producer shared per process. > > > > 1 million records in a week is not very many records, it works down to > ~1.6 > > records a second on average, which is nothing (we typically see 1 > million+ > > messages per second on our clusters). Or maybe your load is spikier than > > that? > > > > Generally if you have multiple producer instances they will fail slightly > > differently, but most failures that hit one (e.g. a broker going down and > > the controller not changing over the leader fast enough). > > > > Thanks > > > > Tom Crayford > > Heroku Kafka > > > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <hafsa.a...@matchinguu.com> > > wrote: > > > > > Hello Folks, > > > > > > I am using Kafka (0.9) in my company and it is expected that we are > going > > > to receive 1 million records in next week. I have many topics for > solely > > > different purposes. Is it good that I define one producer per topic or > > > create one producer for every topic? > > > > > > Right now, I have only 4 topics and each one is expected to receive 1 > > > million record in next week and after 4 months, we will receive 10 > > million > > > records. > > > > > > > > > Is it possible in Kafka that if one producer fails, then other producer > > > also does not work? Please also suggest the safe strategy to go. > > > > > > Best Regards, > > > Hafsa > > > > > >
