Re: use page cache as much as possiblee

Thank you Kishore, I see that the end-to-end latency may not be reduced by resetting the flush time manually. But if the default flush.ms is Long.Max_Value, why I see the disk usage of the brokers constantly increasing when the producer is pushing in data? Should that be happen once a while? The o

Re: use page cache as much as possiblee

Actually in 0.8.2, flush.ms & flush.messages are recommended to be left defaults (Long.MAX_VALUE) http://kafka.apache.org/documentation.html (search for flush.ms) The disk flush and the committed offset are two independent things. As long as you have replication, the recommended thing is to leave

Re: 0.8.2 producer and single message requests

Hi Neelesh :) The new producer has configuration for controlling the batch sizes. By default, it will batch as much as possible without delay (controlled by linger.ms) and without using too much memory (controlled by batch.size). As mentioned in the docs, you can set batch.size to 0 to disable ba

0.8.2 producer and single message requests

We are fronting all our Kafka requests with a simple web service (we do some additional massaging and writing to other stores as well). The new KafkaProducer in 0.8.2 seems very geared towards producer batching. Most of our payload are single messages. Producer batching basically sets us up for lo

Re: Message corruption with new Java client + snappy + broker restart

I am also seeing this issue when using the new producer and snappy compression, running mirrormaker (trunk, aug 10 or so). I'm using snappy 1.1.1.7 [2015-08-14 14:15:27,876] WARN Got error produce response with correlation id 5151552 on topic-partition mytopic-56, retrying (2147480801 attempts lef

Re: Kafka java consumer

There's not a precise date for the release, ~1.5 or 2 months from now. On Fri, Aug 14, 2015 at 3:45 PM, Abhijith Prabhakar wrote: > Thanks Ewen. Any idea when we can expect 0.8.3? > > > > On Aug 14, 2015, at 5:36 PM, Ewen Cheslack-Postava > wrote: > > > > Hi Abhijith, > > > > You should be usi

Re: Kafka java consumer

Thanks Ewen. Any idea when we can expect 0.8.3? > On Aug 14, 2015, at 5:36 PM, Ewen Cheslack-Postava wrote: > > Hi Abhijith, > > You should be using KafkaProducer, but KafkaConsumer is not ready yet. The > APIs are included in 0.8.2.1, but the implementation is not ready. Until > 0.8.3 is rel

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Thanks. Any idea when we can expect 0.8.3? Sent from my phone > On Aug 14, 2015, at 5:39 PM, Guozhang Wang wrote: > > +1 for both KAFKA-2189 and 2308. > >> On Fri, Aug 14, 2015 at 7:03 AM, Gwen Shapira wrote: >> >> Will be nice to include Kafka-2308 and fix two critical snappy issues in >>

Re: [DISCUSSION] Kafka 0.8.2.2 release?

+1 for both KAFKA-2189 and 2308. On Fri, Aug 14, 2015 at 7:03 AM, Gwen Shapira wrote: > Will be nice to include Kafka-2308 and fix two critical snappy issues in > the maintenance release. > > Gwen > On Aug 14, 2015 6:16 AM, "Grant Henke" wrote: > > > Just to clarify. Will KAFKA-2189 be the only

Re: Kafka java consumer

Hi Abhijith, You should be using KafkaProducer, but KafkaConsumer is not ready yet. The APIs are included in 0.8.2.1, but the implementation is not ready. Until 0.8.3 is released, you cannot rely only on kafka-clients if you want to write a consumer. You'll need to depend on the main kafka jar and

Kafka java consumer

Hi All, I am newbie to Kafka and was looking to use java client implementation org.apache.kafka:kafka-clients:0.8.2.1. I was trying to write a consumer group using example given here: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example

Re: Help with SocketTimeoutException while reading from Kafka cluster

On Wednesday 12 August 2015 04:59 AM, venkatesh kavuluri wrote: 83799 [c3-onboard_-2-9571-1439334326956-cfa8b46a-leader-finder-thread] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: java.net.SocketTimeoutException 163931 [c3-onboard_-2-9571-1439334326956-cfa8b46a-l

Re: Undecipherable error in zookeeper on initial connection

Such "errors" are very typical in zookeeper logs - it's very noisy. I typically ignore those errors and try and debug the Kafka issue either via Kafka logs, Kafka thread dumps and/or zookeeper shell. Anyway, how are you adding the topics (script, code?) and what exactly are you noticing? Runni

Re: use page cache as much as possiblee

So if I understand correctly, even if I delay flushing, the consumer will get the messages as soon as the broker receives them and put them into page cache (assuming producer doesn't wait for acks from brokers)? And will the decrease of log.flush interval help reduce latency between producer and c

Re: use page cache as much as possiblee

Thank you Gwen for correcting me. This document ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication) in "Writes" section also has specified the same thing as you have mentioned. One thing is not clear to me as to what happens when the Replicas add the message to memory but the lea

Re: To Synch a failed broker?

Juts to answer my own question,I found the source problem. It was because the brokers couldn't communicate with each other. By opening TCP ports (in my case EC2 security policy) the problem is solved. On Wed, Aug 12, 2015 at 4:32 PM, shahab wrote: > Sorry for posting the email again, because

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Will be nice to include Kafka-2308 and fix two critical snappy issues in the maintenance release. Gwen On Aug 14, 2015 6:16 AM, "Grant Henke" wrote: > Just to clarify. Will KAFKA-2189 be the only patch in the release? > > On Fri, Aug 14, 2015 at 7:35 AM, Manikumar Reddy > wrote: > > > +1 for 0

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Just to clarify. Will KAFKA-2189 be the only patch in the release? On Fri, Aug 14, 2015 at 7:35 AM, Manikumar Reddy wrote: > +1 for 0.8.2.2 release > > On Fri, Aug 14, 2015 at 5:49 PM, Ismael Juma wrote: > > > I think this is a good idea as the change is minimal on our side and it > has > > be

Re: [DISCUSSION] Kafka 0.8.2.2 release?

+1 for 0.8.2.2 release On Fri, Aug 14, 2015 at 5:49 PM, Ismael Juma wrote: > I think this is a good idea as the change is minimal on our side and it has > been tested in production for some time by the reporter. > > Best, > Ismael > > On Fri, Aug 14, 2015 at 1:15 PM, Jun Rao wrote: > > > Hi, E

Re: [DISCUSSION] Kafka 0.8.2.2 release?

I think this is a good idea as the change is minimal on our side and it has been tested in production for some time by the reporter. Best, Ismael On Fri, Aug 14, 2015 at 1:15 PM, Jun Rao wrote: > Hi, Everyone, > > Since the release of Kafka 0.8.2.1, a number of people have reported an > issue w

[DISCUSSION] Kafka 0.8.2.2 release?

Hi, Everyone, Since the release of Kafka 0.8.2.1, a number of people have reported an issue with snappy compression ( https://issues.apache.org/jira/browse/KAFKA-2189). Basically, if they use snappy in 0.8.2.1, they will experience a 2-3X space increase. The issue has since been fixed in trunk (ju

Re: 0.8.2.1 upgrade causes much more IO

Hi, Andrew, Yes, I agree that this is a serious issue. Let me start a discussion thread on this to see if there is any objection in doing an 0.8.2.2 release just for this. Thanks, Jun On Thu, Aug 13, 2015 at 1:10 PM, Andrew Otto wrote: > Hey all, > > Just wanted to confirm, this was totally o