Is that "one instance of the producer class per topic"? I'd recommend just having a single producer shared per process.
1 million records in a week is not very many records, it works down to ~1.6 records a second on average, which is nothing (we typically see 1 million+ messages per second on our clusters). Or maybe your load is spikier than that? Generally if you have multiple producer instances they will fail slightly differently, but most failures that hit one (e.g. a broker going down and the controller not changing over the leader fast enough). Thanks Tom Crayford Heroku Kafka On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <hafsa.a...@matchinguu.com> wrote: > Hello Folks, > > I am using Kafka (0.9) in my company and it is expected that we are going > to receive 1 million records in next week. I have many topics for solely > different purposes. Is it good that I define one producer per topic or > create one producer for every topic? > > Right now, I have only 4 topics and each one is expected to receive 1 > million record in next week and after 4 months, we will receive 10 million > records. > > > Is it possible in Kafka that if one producer fails, then other producer > also does not work? Please also suggest the safe strategy to go. > > Best Regards, > Hafsa >
