Aha, yep that helped a lot.
One producer per process. There's not really a per producer topic limit.
There's buffering and batching space, but assuming you have sufficient
memory (which is by the partition, not by topic), you'll be fine.
Thanks
Tom Crayford
Heroku Kafka
On Tue, May 24, 2016 at 4:46 PM, Hafsa Asif <hafsa.a...@matchinguu.com>
wrote:
> One more question:
> How many topics can be easily handled by one producer?
>
> Hafsa
>
> 2016-05-24 17:39 GMT+02:00 Hafsa Asif <hafsa.a...@matchinguu.com>:
>
> > Ok, let me rephrase (may be I am not using correct terms):
> > Simply consider I have 2 topics, and I have both Java and NodeJS client
> > for Kafka.
> >
> > *NodeJS:*
> > Is it good that I write two producers per each topic like that :
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *var producer1 = new Producer(client);producer1.on('ready', function ()
> > {});producer1.on('error', function (err) { });producer1.send(payloads,
> > cb);var producer2 = new Producer(client);producer2.on('ready', function
> ()
> > {});producer2.on('error', function (err) { });producer2.send(payloads,
> > cb);*
> > Or, I should create one producer for all 2 topics.
> >
> >
> >
> > *Java:*Is it good that I write two producers per each topic like that :
> >
> >
> >
> >
> > *private static Producer<Integer, String> producer1;private static
> > Producer<Integer, String> producer2;producer1 = new Producer<>(new
> > ProducerConfig(properties));producer2 = new Producer<>(new
> > ProducerConfig(properties));*
> >
> > Or, I should create one producer for all 2 topics.
> >
> > Suggest your answer in the light of my estimations (10 topics and 1
> > million records per each topic in next week)
> >
> > Best Regards,
> > Hafsa
> >
> >
> >
> >
> >
> > 2016-05-24 17:23 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> >
> >> Hi,
> >>
> >> I think I'm a bit confused. When you say "one producer per topic", do
> you
> >> mean one instance of the JVM application that's producing per topic?
> >>
> >> Thanks
> >>
> >> Tom
> >>
> >> On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif <hafsa.a...@matchinguu.com>
> >> wrote:
> >>
> >> > Tom,
> >> >
> >> > Thank you for your answer. No, I am talking about one PRODUCER for
> each
> >> > topic, not one instance of same producer class. I am asking for
> general
> >> > concept only.
> >> > Actually we are just growing and not so much far from the case of 1
> >> > million records per sec. Just considering our future case, I need your
> >> > suggestion in more detail, that in general is it a good practice to:
> >> > 1. Prepare a single producer for multiple topics (consider 10 topics)
> .
> >> > 2. Prepare 10 producers for 10 topics respectively.
> >> >
> >> > Your answer is quite satisfying for me, but I need more details so
> that
> >> I
> >> > can convince my team in a good way.
> >> >
> >> > Best Regards,
> >> > Hafsa
> >> >
> >> > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> >> >
> >> > > Is that "one instance of the producer class per topic"? I'd
> recommend
> >> > just
> >> > > having a single producer shared per process.
> >> > >
> >> > > 1 million records in a week is not very many records, it works down
> to
> >> > ~1.6
> >> > > records a second on average, which is nothing (we typically see 1
> >> > million+
> >> > > messages per second on our clusters). Or maybe your load is spikier
> >> than
> >> > > that?
> >> > >
> >> > > Generally if you have multiple producer instances they will fail
> >> slightly
> >> > > differently, but most failures that hit one (e.g. a broker going
> down
> >> and
> >> > > the controller not changing over the leader fast enough).
> >> > >
> >> > > Thanks
> >> > >
> >> > > Tom Crayford
> >> > > Heroku Kafka
> >> > >
> >> > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <
> >> hafsa.a...@matchinguu.com>
> >> > > wrote:
> >> > >
> >> > > > Hello Folks,
> >> > > >
> >> > > > I am using Kafka (0.9) in my company and it is expected that we
> are
> >> > going
> >> > > > to receive 1 million records in next week. I have many topics for
> >> > solely
> >> > > > different purposes. Is it good that I define one producer per
> topic
> >> or
> >> > > > create one producer for every topic?
> >> > > >
> >> > > > Right now, I have only 4 topics and each one is expected to
> receive
> >> 1
> >> > > > million record in next week and after 4 months, we will receive 10
> >> > > million
> >> > > > records.
> >> > > >
> >> > > >
> >> > > > Is it possible in Kafka that if one producer fails, then other
> >> producer
> >> > > > also does not work? Please also suggest the safe strategy to go.
> >> > > >
> >> > > > Best Regards,
> >> > > > Hafsa
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
