Ok, let me rephrase (may be I am not using correct terms):
Simply consider I have 2 topics, and I have both Java and NodeJS client for
Kafka.
*NodeJS:*
Is it good that I write two producers per each topic like that :
*var producer1 = new Producer(client);producer1.on('ready', function ()
{});producer1.on('error', function (err) { });producer1.send(payloads,
cb);var producer2 = new Producer(client);producer2.on('ready', function ()
{});producer2.on('error', function (err) { });producer2.send(payloads,
cb);*
Or, I should create one producer for all 2 topics.
*Java:*Is it good that I write two producers per each topic like that :
*private static Producer<Integer, String> producer1;private static
Producer<Integer, String> producer2;producer1 = new Producer<>(new
ProducerConfig(properties));producer2 = new Producer<>(new
ProducerConfig(properties));*
Or, I should create one producer for all 2 topics.
Suggest your answer in the light of my estimations (10 topics and 1 million
records per each topic in next week)
Best Regards,
Hafsa
2016-05-24 17:23 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> Hi,
>
> I think I'm a bit confused. When you say "one producer per topic", do you
> mean one instance of the JVM application that's producing per topic?
>
> Thanks
>
> Tom
>
> On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif <hafsa.a...@matchinguu.com>
> wrote:
>
> > Tom,
> >
> > Thank you for your answer. No, I am talking about one PRODUCER for each
> > topic, not one instance of same producer class. I am asking for general
> > concept only.
> > Actually we are just growing and not so much far from the case of 1
> > million records per sec. Just considering our future case, I need your
> > suggestion in more detail, that in general is it a good practice to:
> > 1. Prepare a single producer for multiple topics (consider 10 topics) .
> > 2. Prepare 10 producers for 10 topics respectively.
> >
> > Your answer is quite satisfying for me, but I need more details so that I
> > can convince my team in a good way.
> >
> > Best Regards,
> > Hafsa
> >
> > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> >
> > > Is that "one instance of the producer class per topic"? I'd recommend
> > just
> > > having a single producer shared per process.
> > >
> > > 1 million records in a week is not very many records, it works down to
> > ~1.6
> > > records a second on average, which is nothing (we typically see 1
> > million+
> > > messages per second on our clusters). Or maybe your load is spikier
> than
> > > that?
> > >
> > > Generally if you have multiple producer instances they will fail
> slightly
> > > differently, but most failures that hit one (e.g. a broker going down
> and
> > > the controller not changing over the leader fast enough).
> > >
> > > Thanks
> > >
> > > Tom Crayford
> > > Heroku Kafka
> > >
> > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <hafsa.a...@matchinguu.com
> >
> > > wrote:
> > >
> > > > Hello Folks,
> > > >
> > > > I am using Kafka (0.9) in my company and it is expected that we are
> > going
> > > > to receive 1 million records in next week. I have many topics for
> > solely
> > > > different purposes. Is it good that I define one producer per topic
> or
> > > > create one producer for every topic?
> > > >
> > > > Right now, I have only 4 topics and each one is expected to receive 1
> > > > million record in next week and after 4 months, we will receive 10
> > > million
> > > > records.
> > > >
> > > >
> > > > Is it possible in Kafka that if one producer fails, then other
> producer
> > > > also does not work? Please also suggest the safe strategy to go.
> > > >
> > > > Best Regards,
> > > > Hafsa
> > > >
> > >
> >
>
