Interesting discussion! What do you mean here by a process? Is that a thread or the JVM process?
On Tue, May 24, 2016 at 5:49 PM, Tom Crayford <tcrayf...@heroku.com> wrote: > Aha, yep that helped a lot. > > One producer per process. There's not really a per producer topic limit. > There's buffering and batching space, but assuming you have sufficient > memory (which is by the partition, not by topic), you'll be fine. > > Thanks > > Tom Crayford > Heroku Kafka > > On Tue, May 24, 2016 at 4:46 PM, Hafsa Asif <hafsa.a...@matchinguu.com> > wrote: > > > One more question: > > How many topics can be easily handled by one producer? > > > > Hafsa > > > > 2016-05-24 17:39 GMT+02:00 Hafsa Asif <hafsa.a...@matchinguu.com>: > > > > > Ok, let me rephrase (may be I am not using correct terms): > > > Simply consider I have 2 topics, and I have both Java and NodeJS client > > > for Kafka. > > > > > > *NodeJS:* > > > Is it good that I write two producers per each topic like that : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *var producer1 = new Producer(client);producer1.on('ready', function () > > > {});producer1.on('error', function (err) { > });producer1.send(payloads, > > > cb);var producer2 = new Producer(client);producer2.on('ready', function > > () > > > {});producer2.on('error', function (err) { > });producer2.send(payloads, > > > cb);* > > > Or, I should create one producer for all 2 topics. > > > > > > > > > > > > *Java:*Is it good that I write two producers per each topic like that : > > > > > > > > > > > > > > > *private static Producer<Integer, String> producer1;private static > > > Producer<Integer, String> producer2;producer1 = new Producer<>(new > > > ProducerConfig(properties));producer2 = new Producer<>(new > > > ProducerConfig(properties));* > > > > > > Or, I should create one producer for all 2 topics. > > > > > > Suggest your answer in the light of my estimations (10 topics and 1 > > > million records per each topic in next week) > > > > > > Best Regards, > > > Hafsa > > > > > > > > > > > > > > > > > > 2016-05-24 17:23 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>: > > > > > >> Hi, > > >> > > >> I think I'm a bit confused. When you say "one producer per topic", do > > you > > >> mean one instance of the JVM application that's producing per topic? > > >> > > >> Thanks > > >> > > >> Tom > > >> > > >> On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif < > hafsa.a...@matchinguu.com> > > >> wrote: > > >> > > >> > Tom, > > >> > > > >> > Thank you for your answer. No, I am talking about one PRODUCER for > > each > > >> > topic, not one instance of same producer class. I am asking for > > general > > >> > concept only. > > >> > Actually we are just growing and not so much far from the case of 1 > > >> > million records per sec. Just considering our future case, I need > your > > >> > suggestion in more detail, that in general is it a good practice to: > > >> > 1. Prepare a single producer for multiple topics (consider 10 > topics) > > . > > >> > 2. Prepare 10 producers for 10 topics respectively. > > >> > > > >> > Your answer is quite satisfying for me, but I need more details so > > that > > >> I > > >> > can convince my team in a good way. > > >> > > > >> > Best Regards, > > >> > Hafsa > > >> > > > >> > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>: > > >> > > > >> > > Is that "one instance of the producer class per topic"? I'd > > recommend > > >> > just > > >> > > having a single producer shared per process. > > >> > > > > >> > > 1 million records in a week is not very many records, it works > down > > to > > >> > ~1.6 > > >> > > records a second on average, which is nothing (we typically see 1 > > >> > million+ > > >> > > messages per second on our clusters). Or maybe your load is > spikier > > >> than > > >> > > that? > > >> > > > > >> > > Generally if you have multiple producer instances they will fail > > >> slightly > > >> > > differently, but most failures that hit one (e.g. a broker going > > down > > >> and > > >> > > the controller not changing over the leader fast enough). > > >> > > > > >> > > Thanks > > >> > > > > >> > > Tom Crayford > > >> > > Heroku Kafka > > >> > > > > >> > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif < > > >> hafsa.a...@matchinguu.com> > > >> > > wrote: > > >> > > > > >> > > > Hello Folks, > > >> > > > > > >> > > > I am using Kafka (0.9) in my company and it is expected that we > > are > > >> > going > > >> > > > to receive 1 million records in next week. I have many topics > for > > >> > solely > > >> > > > different purposes. Is it good that I define one producer per > > topic > > >> or > > >> > > > create one producer for every topic? > > >> > > > > > >> > > > Right now, I have only 4 topics and each one is expected to > > receive > > >> 1 > > >> > > > million record in next week and after 4 months, we will receive > 10 > > >> > > million > > >> > > > records. > > >> > > > > > >> > > > > > >> > > > Is it possible in Kafka that if one producer fails, then other > > >> producer > > >> > > > also does not work? Please also suggest the safe strategy to go. > > >> > > > > > >> > > > Best Regards, > > >> > > > Hafsa > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >