Hi Frank,
Here are the explicitly set properties:
Consumer:
Properties props = new Properties();
props.put("zk.connect", KafkaProperties.zkConnect);
props.put("groupid", KafkaProperties.groupId);
props.put("zk.sessiontimeout.ms", "400");
props.put("zk.synctime.ms", "200");
Hi Francis/Stone,
I have modified log.default.flush.interval.ms to have a value of 1 in
config/server.properties file. The time did not come down as can be seen
from the following output:
Message received at 2013-05-17 11:54:02.973: Message produced at 2013-05-17
11:54:01.444: Message_1
Message r
+1 ... very good suggestion.
Sent from Samsung Mobile
Original message
From: Stone
Date: 05-17-2013 2:16 AM (GMT-05:00)
To: users@kafka.apache.org
Subject: Re: Kafka performance
Hi Kishore,
Since a message is only exposed to the consumers after it is flushed. Maybe
you ca
What are the settings you have set on the producer ... is there any sync ack ?
If so which type ?
In any case, it would be useful to know the parameters you set on the consumer
and producer.
Thanks
Frank
Sent from Samsung Mobile
Original message
From: "Kishore V. Kopalle
Hi Kishore,
Since a message is only exposed to the consumers after it is flushed. Maybe
you can try lower log.default.flush.interval.ms which controls the maximum
time that a message in any topic is kept in memory before flushed to disk (
by default it's 3000ms )
ref - http://kafka.apache.org/07/
Hi Frank,
Yes, they are on the same machine. Zoo Keeper, Kafka server and
the KafkaConsumerProducerDemo (with one producer and one consumer) are
running on the same computer.
Regards,
Kishore
On Fri, May 17, 2013 at 11:38 AM, Francis Dallaire <
francis.dalla...@ubisoft.com> wrote:
> Quick ques
Quick question. Are the producer and consumer on the same machine ? If not, are
you accounting for any difference in the time setting between producer and
consumer ?
Frank
Sent from Samsung Mobile
Original message
From: Kishore Kopalle
Date: 05-17-2013 1:41 AM (GMT-05:00)
Hello All,
In my earlier mail, I am sorry I did not give lot of details on my setup
where I am measuring performance between producer and consumer. Here are
the details:
- I am using Kafka version 0.7 downloaded from Apache web site.
- I am using a modified version of KafkaConsumerProducerDemo e
Hello All,
In my earlier mail, I am sorry I did not give lot of details on my setup
where I am measuring performance between producer and consumer. Here are
the details:
- I am using Kafka version 0.7 downloaded from Apache web site.
- I am using a modified version of KafkaConsumerProducerDemo e
Can a request be made to zookeeper for this feature?
Thanks,
rob
> -Original Message-
> From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
> Sent: Thursday, May 16, 2013 9:53 PM
> To: users@kafka.apache.org
> Subject: Re: are commitOffsets botched to zookeeper?
>
> Currently Kafka depe
We want to ensure only-once message processing, but we also want the benefit
of rebalancing. commitOffsets updates all partitions from out of a
connector instance. We want to commit the offset for just the partition
that delivered a message to the iterator, even if several fetchers are
feeding a
HI
I have a 3 node kafka cluster, and i have 2 topics( one of very high
priority and other normal data), i need to transmit the normal data to two
brokers in the cluster , and the high priority data directly to the 3rd
broker.
This is so that my high priority data has a clear path and can be
tran
Hi
I was just doing some benchmarking with a 3node cluster. If one broker goes
down , what happens? Does the producer go down ? that's what happened in my
case.
Is the data lost? Or does it get distributed amongst the other brokers?
--
Chitra Raveendran
Data Scientist
*Flutura | Decision Scienc
Could the producer be adapted to support the interface of the consumer?
Thanks,
rob
> -Original Message-
> From: Jun Rao [mailto:jun...@gmail.com]
> Sent: Thursday, May 16, 2013 10:04 PM
> To: users@kafka.apache.org
> Subject: Re: could an Encoder/Decoder be stateful?
>
> The encoder ins
Immediately monitoring. Later, possible thresholding to evoke a
reconfiguration of the number of partitions into a new topic and migrate
message index/logs and redirect pubs/subs to the new topic, during a traffic
spike.
> -Original Message-
> From: Jun Rao [mailto:jun...@gmail.com]
> Sen
Hello All,
I am evaluating Kafka for use in a real time financial product. I am tying
to calculate the time it takes for a message to be fetched by the consumer
after it has been sent by a producer. I am finding that this takes a long
time of between 900-1200 ms per message on Windows 7 single cor
The encoder instance can't be shared in different producers since it's
instantiated through java reflection. Decoder instance can actually be
shared by different consumers since it's passed in directly through the
consumer api.
Thanks,
Jun
On Thu, May 16, 2013 at 5:58 AM, Withers, Robert wrote:
Can you describe your requirements in a little more detail?
Thanks,
Neha
On May 16, 2013 6:11 AM, "Withers, Robert" wrote:
> is it technically feasible to use an only-once simple consumer within a
> consumer group?
>
> thanks,
> rob
Can you share the consumer log4j files during the time it rebalances?
Thanks,
Neha
On May 16, 2013 10:28 AM, "Javier Iseruk"
wrote:
> Hi.
> I have a consumer group with five consumers, but only three are consuming.
> The topic has 25 partitions (5 brokers with 5 partitions each one), so it
> sho
It seems that the zookeeper host string is null or empty. Can you paste the
code you used to instantiate the consumer?
Thanks,
Neha
On May 16, 2013 12:45 PM, "Seshadri, Balaji"
wrote:
> Hi,
>
> I'm trying to write webMethods consumer for Kafka and I get the following
> error when I try to integr
What do you plan to use this information for?
Thanks,
Jun
On Thu, May 16, 2013 at 5:57 AM, Withers, Robert wrote:
> Any possibility for a queuedMessageCount(topic, partitionNumber) protocol?
>
> thanks,
> rob
>
> From: Jun Rao [jun...@gmail.com]
> Sent:
Currently Kafka depends on zookeeper 3.3.4 that doesn't have a batch write
api. So if you commit after every message at a high rate, it will be slow
and inefficient. Besides it will cause zookeeper performance to degrade.
Thanks,
Neha
On May 16, 2013 6:54 PM, "Rob Withers" wrote:
> We are callin
We are calling commitOffsets after every message consumption. It looks to be
~60% slower, with 29 partitions. If a single KafkaStream thread is from a
connector, and there are 29 partitions, then commitOffsets sends 29 offset
updates, correct? Are these offset updates batched in one send to z
Hi,
I'm trying to write webMethods consumer for Kafka and I get the following error
when I try to integrate.
java.lang.NullPointerException
at org.apache.zookeeper.ClientCnxn.(ClientCnxn.java:360)
at org.apache.zookeeper.ClientCnxn.(ClientCnxn.java:331)
Hi.
I have a consumer group with five consumers, but only three are consuming.
The topic has 25 partitions (5 brokers with 5 partitions each one), so it
shouldn't be the problem the partitions quantity.
In zookeeper, I see the five consumers in /consumers/$GROUP/ids, but in
/consumers/$GROUP/owner
is it technically feasible to use an only-once simple consumer within a
consumer group?
thanks,
rob
Excellent news. Thanks, Neha.
From: Neha Narkhede [neha.narkh...@gmail.com]
Sent: Wednesday, May 15, 2013 10:25 PM
To: users@kafka.apache.org
Subject: Re: when do you think 0.8 could get promoted from beta to a release?
We are currently working on fixing a
Is there a way to share an encoder instance, or inject shared state, across
producers/consumers? Guice?
From: Jun Rao [jun...@gmail.com]
Sent: Wednesday, May 15, 2013 11:02 PM
To: users@kafka.apache.org
Subject: Re: could an Encoder/Decoder be stateful?
E
Any possibility for a queuedMessageCount(topic, partitionNumber) protocol?
thanks,
rob
From: Jun Rao [jun...@gmail.com]
Sent: Wednesday, May 15, 2013 10:59 PM
To: users@kafka.apache.org
Subject: Re: API to to query messages amount under one topic
In 0.8, y
29 matches
Mail list logo
