Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-27 Thread tao xiao
No. With receive.buffer.bytes setting to 64K I am unable to reproduce the error. BTW I set the message size to 10K when testing with 64K buffer size On Thu, 28 Jan 2016 at 01:35 Jason Gustafson wrote: > Hey Tao, > > If you increase "receive.buffer.bytes" to 64K, can you still reproduce the > pro

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-27 Thread Jason Gustafson
Hey Tao, If you increase "receive.buffer.bytes" to 64K, can you still reproduce the problem? -Jason On Tue, Jan 26, 2016 at 11:18 PM, Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.pl> wrote: > Jason, > > My os/vm is OSX 10.11.3, JDK 1.8.0.40 > > — > Krzysztof > On 26 January 2016 at

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-26 Thread Krzysztof Ciesielski
Jason, My os/vm is OSX 10.11.3, JDK 1.8.0.40 —  Krzysztof On 26 January 2016 at 19:04:58, Jason Gustafson (ja...@confluent.io) wrote: Hey Krzysztof, So far I haven't had any luck figuring out the cause of the 5 second pause, but I've reproduced it with the old consumer on 0.8.2, so that rul

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-26 Thread tao xiao
I managed to reproduce this issue on my mac with receive.buffer.bytes setting to new consumer default value. My JVM is hotspot 64 bit 1.7.0_60 and mac 10.10.5 On Wed, 27 Jan 2016 at 02:54 Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.pl> wrote: > Hi Jason, > > Lowering "receive.buffer.

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-26 Thread Krzysztof Ciesielski
Hi Jason, Lowering "receive.buffer.bytes” helps, but when the message size gets bigger - it comes back again. I will test with 65536 and check how big the message has to be to make the issue reappear with this value (I suspect that quite big). --  Krzysztof Ciesielski SoftwareMill On 26 Januar

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-26 Thread Jason Gustafson
Hey Krzysztof, So far I haven't had any luck figuring out the cause of the 5 second pause, but I've reproduced it with the old consumer on 0.8.2, so that rules out anything specific to the new consumer. Can you tell me which os/jvm you're seeing it with? Also, can you try changing the "receive.buf

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-24 Thread Ismael Juma
Hi, On Sun, Jan 24, 2016 at 7:17 PM, Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.com> wrote: > Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA > Jason already did so: https://issues.apache.org/jira/browse/KAFKA-3135 Feel free to add relevant information there.

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-24 Thread Krzysztof Ciesielski
Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA On Jan 22, 2016 7:24 AM, "Jason Gustafson" wrote: > Hi Krzysztof, > > This is definitely weird. I see the data in the broker's send queue, but > there's a delay of 5 seconds before it's sent to the client. Can you create > a JIR

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-22 Thread Jason Gustafson
I see the pause regardless of the client's poll timeout. I thought it might be caused by the quota manager (which is supposed to delay fetches if a quota is exceeded), but this is looking more like a problem in the server's network layer. I went ahead and opened KAFKA-3135, so we can move investiga

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-21 Thread Samya Maiti
Providing a non-Zero value (Zero is default), for public ConsumerRecords poll(long timeout) Works fine for me with no gaps,but said so is just a work around. Consumer is definitely picking up messages with some delay. -Sam > On 22-Jan-2016, at 11:54 am, Jason Gustafson wrote: > >

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-21 Thread Jason Gustafson
Hi Krzysztof, This is definitely weird. I see the data in the broker's send queue, but there's a delay of 5 seconds before it's sent to the client. Can you create a JIRA? Thanks, Jason On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti wrote: > +1, facing same issue. > -Sam > > On 22-Jan-2016, at

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-21 Thread Samya Maiti
+1, facing same issue. -Sam > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski > wrote: > > Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. > Here's a runnable gist illustrating the problem: > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on >

Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-21 Thread Krzysztof Ciesielski
Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. Here's a runnable gist illustrating the problem: https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on localhost:9092) Scenario description: First, a producer writes 50 elements into a topic Then, a