One more question:
How many topics can be easily handled by one producer?
Hafsa
2016-05-24 17:39 GMT+02:00 Hafsa Asif <hafsa.a...@matchinguu.com>:
> Ok, let me rephrase (may be I am not using correct terms):
> Simply consider I have 2 topics, and I have both Java and NodeJS client
> for Kafka.
>
> *NodeJS:*
> Is it good that I write two producers per each topic like that :
>
>
>
>
>
>
>
>
>
>
>
>
> *var producer1 = new Producer(client);producer1.on('ready', function ()
> {});producer1.on('error', function (err) { });producer1.send(payloads,
> cb);var producer2 = new Producer(client);producer2.on('ready', function ()
> {});producer2.on('error', function (err) { });producer2.send(payloads,
> cb);*
> Or, I should create one producer for all 2 topics.
>
>
>
> *Java:*Is it good that I write two producers per each topic like that :
>
>
>
>
> *private static Producer<Integer, String> producer1;private static
> Producer<Integer, String> producer2;producer1 = new Producer<>(new
> ProducerConfig(properties));producer2 = new Producer<>(new
> ProducerConfig(properties));*
>
> Or, I should create one producer for all 2 topics.
>
> Suggest your answer in the light of my estimations (10 topics and 1
> million records per each topic in next week)
>
> Best Regards,
> Hafsa
>
>
>
>
>
> 2016-05-24 17:23 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
>
>> Hi,
>>
>> I think I'm a bit confused. When you say "one producer per topic", do you
>> mean one instance of the JVM application that's producing per topic?
>>
>> Thanks
>>
>> Tom
>>
>> On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif <hafsa.a...@matchinguu.com>
>> wrote:
>>
>> > Tom,
>> >
>> > Thank you for your answer. No, I am talking about one PRODUCER for each
>> > topic, not one instance of same producer class. I am asking for general
>> > concept only.
>> > Actually we are just growing and not so much far from the case of 1
>> > million records per sec. Just considering our future case, I need your
>> > suggestion in more detail, that in general is it a good practice to:
>> > 1. Prepare a single producer for multiple topics (consider 10 topics) .
>> > 2. Prepare 10 producers for 10 topics respectively.
>> >
>> > Your answer is quite satisfying for me, but I need more details so that
>> I
>> > can convince my team in a good way.
>> >
>> > Best Regards,
>> > Hafsa
>> >
>> > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
>> >
>> > > Is that "one instance of the producer class per topic"? I'd recommend
>> > just
>> > > having a single producer shared per process.
>> > >
>> > > 1 million records in a week is not very many records, it works down to
>> > ~1.6
>> > > records a second on average, which is nothing (we typically see 1
>> > million+
>> > > messages per second on our clusters). Or maybe your load is spikier
>> than
>> > > that?
>> > >
>> > > Generally if you have multiple producer instances they will fail
>> slightly
>> > > differently, but most failures that hit one (e.g. a broker going down
>> and
>> > > the controller not changing over the leader fast enough).
>> > >
>> > > Thanks
>> > >
>> > > Tom Crayford
>> > > Heroku Kafka
>> > >
>> > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <
>> hafsa.a...@matchinguu.com>
>> > > wrote:
>> > >
>> > > > Hello Folks,
>> > > >
>> > > > I am using Kafka (0.9) in my company and it is expected that we are
>> > going
>> > > > to receive 1 million records in next week. I have many topics for
>> > solely
>> > > > different purposes. Is it good that I define one producer per topic
>> or
>> > > > create one producer for every topic?
>> > > >
>> > > > Right now, I have only 4 topics and each one is expected to receive
>> 1
>> > > > million record in next week and after 4 months, we will receive 10
>> > > million
>> > > > records.
>> > > >
>> > > >
>> > > > Is it possible in Kafka that if one producer fails, then other
>> producer
>> > > > also does not work? Please also suggest the safe strategy to go.
>> > > >
>> > > > Best Regards,
>> > > > Hafsa
>> > > >
>> > >
>> >
>>
>
>
