Sorry to bump this up, can anyone provide some input on this ? I need to make a
call soon whether kafka is a good fit to our requirement.
On Tuesday, October 4, 2016 8:57 PM, Shamik Banerjee
wrote:
Hi,
I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation
primarily is
Hi,
I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation
primarily is to improve performance where kafka would eliminate the constraint
of pulling 10 messages at a time with a cap of 256kb. Here's a high-level
scenario of my use case. I've a bunch of crawlers which are s
Kafka also collects very useful metrics on request times and their
breakdown.
They are under kafka.network.
On Mon, Sep 14, 2015 at 6:59 AM, Rahul Jain wrote:
> Have you checked the consumer lag? You can use the offset checker tool to
> see if there is a lag.
> On 14 Sep 2015 18:36, "noah" wr
Have you checked the consumer lag? You can use the offset checker tool to
see if there is a lag.
On 14 Sep 2015 18:36, "noah" wrote:
> We're using 0.8.2.1 processing maybe 1 million messages per hour. Each
> message includes tracking information with a timestamp for when it was
> produced, and a
We're using 0.8.2.1 processing maybe 1 million messages per hour. Each
message includes tracking information with a timestamp for when it was
produced, and a timestamp for when it was consumed, to give us roughly the
amount of time it spent in Kafka. On average this number is in the seconds
and ou
By increasing partitions and using kafka from master branch I was able to
cut down the response times into half. But it still seems high and it looks
like there still is a delay between a successful post and the first time
message is seen by the consumers. There are plenty of resources available.
the server.properties file doesn't have all the properties. You can add it
there and try your test.
On Wed, Oct 22, 2014 at 11:41 AM, Mohit Anchlia
wrote:
> I can't find this property in server.properties file. Is that the right
> place to set this parameter?
> On Tue, Oct 21, 2014 at 6:27 PM, J
I can't find this property in server.properties file. Is that the right
place to set this parameter?
On Tue, Oct 21, 2014 at 6:27 PM, Jun Rao wrote:
> Could you also set replica.fetch.wait.max.ms in the broker to sth much
> smaller?
>
> Thanks,
>
> Jun
>
> On Tue, Oct 21, 2014 at 2:15 PM, Mohit A
Could you also set replica.fetch.wait.max.ms in the broker to sth much
smaller?
Thanks,
Jun
On Tue, Oct 21, 2014 at 2:15 PM, Mohit Anchlia
wrote:
> I set the property to 1 in the consumer code that is passed to
> "createJavaConsumerConnector"
> code, but it didn't seem to help
>
> props.put("f
Most of the consumer threads seems to be waiting:
"ConsumerFetcherThread-groupA_ip-10-38-19-230-1413925671158-3cc3e22f-0-0"
prio=10 tid=0x7f0aa84db800 nid=0x5be9 runnable [0x7f0a5a618000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
I set the property to 1 in the consumer code that is passed to
"createJavaConsumerConnector"
code, but it didn't seem to help
props.put("fetch.wait.max.ms", fetchMaxWait);
On Tue, Oct 21, 2014 at 1:21 PM, Guozhang Wang wrote:
> This is a consumer config:
>
> fetch.wait.max.ms
>
> On Tue, Oct 21
This is a consumer config:
fetch.wait.max.ms
On Tue, Oct 21, 2014 at 11:39 AM, Mohit Anchlia
wrote:
> Is this a parameter I need to set it in kafka server or on the client side?
> Also, can you help point out which one exactly is consumer max wait time
> from this list?
>
> https://kafka.apache
Is this a parameter I need to set it in kafka server or on the client side?
Also, can you help point out which one exactly is consumer max wait time
from this list?
https://kafka.apache.org/08/configuration.html
On Tue, Oct 21, 2014 at 11:35 AM, Jay Kreps wrote:
> There was a bug that could lea
There was a bug that could lead to the fetch request from the consumer
hitting it's timeout instead of being immediately triggered by the produce
request. To see if you are effected by that set you consumer max wait time
to 1 ms and see if the latency drops to 1 ms (or, alternately, try with
trunk
This is the version I am using: kafka_2.10-0.8.1.1
I think this is fairly recent version
On Tue, Oct 21, 2014 at 10:57 AM, Jay Kreps wrote:
> What version of Kafka is this? Can you try the same test against trunk? We
> fixed a couple of latency related bugs which may be the cause.
>
> -Jay
>
> O
What version of Kafka is this? Can you try the same test against trunk? We
fixed a couple of latency related bugs which may be the cause.
-Jay
On Tue, Oct 21, 2014 at 10:50 AM, Mohit Anchlia
wrote:
> It's consistently close to 100ms which makes me believe that there are some
> settings that I m
It's consistently close to 100ms which makes me believe that there are some
settings that I might have to tweak, however, I am not sure how to confirm
that assumption :)
On Tue, Oct 21, 2014 at 8:53 AM, Mohit Anchlia
wrote:
> I have a java test that produces messages and then consumer consumers i
I have a java test that produces messages and then consumer consumers it.
Consumers are active all the time. There is 1 consumer for 1 producer. I am
measuring the time between the message is successfully written to the queue
and the time consumer picks it up.
On Tue, Oct 21, 2014 at 8:32 AM, Neha
Can you give more information about the performance test? Which test? Which
queue? How did you measure the dequeue latency.
On Mon, Oct 20, 2014 at 5:09 PM, Mohit Anchlia
wrote:
> I am running a performance test and from what I am seeing is that messages
> are taking about 100ms to pop from the
I am running a performance test and from what I am seeing is that messages
are taking about 100ms to pop from the queue itself and hence making the
test slow. I am looking for pointers of how I can troubleshoot this issue.
There seems to be plenty of CPU and IO available. I am running 22 producers
20 matches
Mail list logo
