Re: High Latency in Kafka

2015-02-10 Thread Andrey Yegorov
-beta and async send. I remember that I had to tune some parameters for kafka producer, increased buffer sizes and something else. HTH. -- Andrey Yegorov On Tue, Feb 10, 2015 at 5:54 AM, Vineet Mishra wrote: > Hi Gwen, > > Well I have gone through this link while trying to set

Re: Best way to replace a single broker

2015-05-14 Thread Andrey Yegorov
election, whatever it is called) if it does not happen automatically. -- Andrey Yegorov On Thu, May 14, 2015 at 11:12 AM, Rajiv Kurian wrote: > Hi all, > > Sometimes we need to replace a kafka broker because it turns out to be a > bad instance. What is the best way of doing this?

KafkaException: Size of FileMessageSet has been truncated during write

2015-05-27 Thread Andrey Yegorov
at kafka.network.Processor.write(SocketServer.scala:375) at kafka.network.Processor.run(SocketServer.scala:247) at java.lang.Thread.run(Thread.java:745) -- Andrey Yegorov

Re: KafkaException: Size of FileMessageSet has been truncated during write

2015-05-28 Thread Andrey Yegorov
Thank you! -- Andrey Yegorov On Wed, May 27, 2015 at 4:42 PM, Jiangjie Qin wrote: > This should be just a message fetch failure. The socket was disconnected > when broker was writing to it. There should not be data loss. > > Jiangjie (Becket) Qin > > On 5/27/15,

Re: compatibility: 0.8.1.1 broker, 0.8.2.2 producer

2015-12-23 Thread Andrey Yegorov
I am using 0.8.2.2 producer with 0.8.1.1 brokers without problems. Version of scala matters if you are building with scala or some other components that use scala. Hope this helps. -- Andrey Yegorov On Wed, Dec 23, 2015 at 1:11 PM, Ewen Cheslack-Postava wrote: > Shlomi, > > Y

Mirroring datacenters without vpn

2014-01-10 Thread Andrey Yegorov
helpful. Do you use vpn between DCs? Where do you run MirrorMaker - in central dc or in remote and why? A lot of question, thank you beforehand for your answers. -- Andrey Yegorov

Re: Mirroring datacenters without vpn

2014-01-16 Thread Andrey Yegorov
Thank you. Reference to KAFKA-1092 is very useful. Unfortunately it is not a part of release version 0.8 but I hope 0.8.1 will see the light soon enough. -- Andrey Yegorov On Fri, Jan 10, 2014 at 5:08 PM, Joel Koshy wrote: > > > > Ops proposed to set up mirror to wo

Re: New Producer Public API

2014-01-24 Thread Andrey Yegorov
not true already - it was not send in case of error.) From that object I can get some info about the error and offset. How d I get the original message back so I can write it to the log? Can you please provide an example? ------ Andrey Yegorov On Fri, Jan 24, 2014 at 11:54 AM, Jay Kreps wrot

Re: New Producer Public API

2014-01-24 Thread Andrey Yegorov
d and reuse it. Since kafka producer already have messages in the batch and knows the batch that failed, it can pass the message to the onError() callback. Am I over-thinking this? -- Andrey Yegorov On Fri, Jan 24, 2014 at 1:15 PM, Jay Kreps wrote: > If I understand your use case

estimating log.retention.bytes

2014-04-22 Thread Andrey Yegorov
ng is: with replication, I'll have 60 'partitions' spread across 3 machines hence 20 per machine. Max space I can allocate per partition is 15TB/20 = 768GB per partition. Am I on the right track? -- Andrey Yegorov

mirrormaker's configuration to minimize/prevent data loss

2014-06-16 Thread Andrey Yegorov
? I am ok with not using async mode on producer, but will it help? Can I configure mirrormaker to exit immediately if producer fails? If this should be a responsibility of an external process, what should I monitor log for to kill the mirroring process in case of error? -- Andrey Yegorov