Interesting discussion!
What do you mean here by a process? Is that a thread or the JVM process?
On Tue, May 24, 2016 at 5:49 PM, Tom Crayford <tcrayf...@heroku.com> wrote:
> Aha, yep that helped a lot.
>
> One producer per process. There's not really a per producer topic limit.
> There's buffering and batching space, but assuming you have sufficient
> memory (which is by the partition, not by topic), you'll be fine.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Tue, May 24, 2016 at 4:46 PM, Hafsa Asif <hafsa.a...@matchinguu.com>
> wrote:
>
> > One more question:
> > How many topics can be easily handled by one producer?
> >
> > Hafsa
> >
> > 2016-05-24 17:39 GMT+02:00 Hafsa Asif <hafsa.a...@matchinguu.com>:
> >
> > > Ok, let me rephrase (may be I am not using correct terms):
> > > Simply consider I have 2 topics, and I have both Java and NodeJS client
> > > for Kafka.
> > >
> > > *NodeJS:*
> > > Is it good that I write two producers per each topic like that :
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *var producer1 = new Producer(client);producer1.on('ready', function ()
> > > {});producer1.on('error', function (err) {
> });producer1.send(payloads,
> > > cb);var producer2 = new Producer(client);producer2.on('ready', function
> > ()
> > > {});producer2.on('error', function (err) {
> });producer2.send(payloads,
> > > cb);*
> > > Or, I should create one producer for all 2 topics.
> > >
> > >
> > >
> > > *Java:*Is it good that I write two producers per each topic like that :
> > >
> > >
> > >
> > >
> > > *private static Producer<Integer, String> producer1;private static
> > > Producer<Integer, String> producer2;producer1 = new Producer<>(new
> > > ProducerConfig(properties));producer2 = new Producer<>(new
> > > ProducerConfig(properties));*
> > >
> > > Or, I should create one producer for all 2 topics.
> > >
> > > Suggest your answer in the light of my estimations (10 topics and 1
> > > million records per each topic in next week)
> > >
> > > Best Regards,
> > > Hafsa
> > >
> > >
> > >
> > >
> > >
> > > 2016-05-24 17:23 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> > >
> > >> Hi,
> > >>
> > >> I think I'm a bit confused. When you say "one producer per topic", do
> > you
> > >> mean one instance of the JVM application that's producing per topic?
> > >>
> > >> Thanks
> > >>
> > >> Tom
> > >>
> > >> On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif <
> hafsa.a...@matchinguu.com>
> > >> wrote:
> > >>
> > >> > Tom,
> > >> >
> > >> > Thank you for your answer. No, I am talking about one PRODUCER for
> > each
> > >> > topic, not one instance of same producer class. I am asking for
> > general
> > >> > concept only.
> > >> > Actually we are just growing and not so much far from the case of 1
> > >> > million records per sec. Just considering our future case, I need
> your
> > >> > suggestion in more detail, that in general is it a good practice to:
> > >> > 1. Prepare a single producer for multiple topics (consider 10
> topics)
> > .
> > >> > 2. Prepare 10 producers for 10 topics respectively.
> > >> >
> > >> > Your answer is quite satisfying for me, but I need more details so
> > that
> > >> I
> > >> > can convince my team in a good way.
> > >> >
> > >> > Best Regards,
> > >> > Hafsa
> > >> >
> > >> > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> > >> >
> > >> > > Is that "one instance of the producer class per topic"? I'd
> > recommend
> > >> > just
> > >> > > having a single producer shared per process.
> > >> > >
> > >> > > 1 million records in a week is not very many records, it works
> down
> > to
> > >> > ~1.6
> > >> > > records a second on average, which is nothing (we typically see 1
> > >> > million+
> > >> > > messages per second on our clusters). Or maybe your load is
> spikier
> > >> than
> > >> > > that?
> > >> > >
> > >> > > Generally if you have multiple producer instances they will fail
> > >> slightly
> > >> > > differently, but most failures that hit one (e.g. a broker going
> > down
> > >> and
> > >> > > the controller not changing over the leader fast enough).
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > Tom Crayford
> > >> > > Heroku Kafka
> > >> > >
> > >> > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <
> > >> hafsa.a...@matchinguu.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hello Folks,
> > >> > > >
> > >> > > > I am using Kafka (0.9) in my company and it is expected that we
> > are
> > >> > going
> > >> > > > to receive 1 million records in next week. I have many topics
> for
> > >> > solely
> > >> > > > different purposes. Is it good that I define one producer per
> > topic
> > >> or
> > >> > > > create one producer for every topic?
> > >> > > >
> > >> > > > Right now, I have only 4 topics and each one is expected to
> > receive
> > >> 1
> > >> > > > million record in next week and after 4 months, we will receive
> 10
> > >> > > million
> > >> > > > records.
> > >> > > >
> > >> > > >
> > >> > > > Is it possible in Kafka that if one producer fails, then other
> > >> producer
> > >> > > > also does not work? Please also suggest the safe strategy to go.
> > >> > > >
> > >> > > > Best Regards,
> > >> > > > Hafsa
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
