Tom, Thank you for your answer. No, I am talking about one PRODUCER for each topic, not one instance of same producer class. I am asking for general concept only. Actually we are just growing and not so much far from the case of 1 million records per sec. Just considering our future case, I need your suggestion in more detail, that in general is it a good practice to: 1. Prepare a single producer for multiple topics (consider 10 topics) . 2. Prepare 10 producers for 10 topics respectively.
Your answer is quite satisfying for me, but I need more details so that I can convince my team in a good way. Best Regards, Hafsa 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>: > Is that "one instance of the producer class per topic"? I'd recommend just > having a single producer shared per process. > > 1 million records in a week is not very many records, it works down to ~1.6 > records a second on average, which is nothing (we typically see 1 million+ > messages per second on our clusters). Or maybe your load is spikier than > that? > > Generally if you have multiple producer instances they will fail slightly > differently, but most failures that hit one (e.g. a broker going down and > the controller not changing over the leader fast enough). > > Thanks > > Tom Crayford > Heroku Kafka > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <hafsa.a...@matchinguu.com> > wrote: > > > Hello Folks, > > > > I am using Kafka (0.9) in my company and it is expected that we are going > > to receive 1 million records in next week. I have many topics for solely > > different purposes. Is it good that I define one producer per topic or > > create one producer for every topic? > > > > Right now, I have only 4 topics and each one is expected to receive 1 > > million record in next week and after 4 months, we will receive 10 > million > > records. > > > > > > Is it possible in Kafka that if one producer fails, then other producer > > also does not work? Please also suggest the safe strategy to go. > > > > Best Regards, > > Hafsa > > >
