Is there a way to change Kafka's data path in Zookeeper transparently to Kafka client?

Hi group, We are running Kafka clusters in production, using Zookeeper without chroot, unwisely. Then there are several Kafka-related dirs in Zookeeper's root path with some general names, such as */consumers*. And recently we have another application coincidently using same */consumers *path, tho

Re: Auto preferred leader elections cause data loss?

cluding 101. > > So if you use the safe settings you should be safe in this scenario. > > Gwen > > > On Sat, Sep 12, 2015 at 3:15 AM, Zhao Weinan wrote: > > > Hi group, > > > > I've been through some data loss on kafka cluster, one case maybe caused >

Re: Auto preferred leader elections cause data loss?

Hi group, I think I've hit the KAFKA-1561 and KAFKA-1211, that is follower's HW is always <= leader's HW, then when leader transits, data loss can happen... So what we could do is try to do every thing to avoid leader transition, right? 2015-09-14 11:59 GMT+08:00 Zhao We

Re: Auto preferred leader elections cause data loss?

n the lag between this new leader and old-gone-leader is lost, right? Thanks! 2015-09-12 18:15 GMT+08:00 Zhao Weinan : > Hi group, > > I've been through some data loss on kafka cluster, one case maybe caused > by the auto preferred leader election. > > Here is the situa

Re: Possible DEAD LOCK for one day at broker controller?

00 Kishore Senji : > Hi Zhao, Do you see any other errors regarding checkpoint file? Is this > reproducible by you and if you can you enable debug log level to get more > info.​ > > On Thu, Aug 20, 2015 at 7:44 AM, Zhao Weinan wrote: > > > Hi Kishore Senji, > > >

Auto preferred leader elections cause data loss?

Hi group, I've been through some data loss on kafka cluster, one case maybe caused by the auto preferred leader election. Here is the situation: 3 brokers = {0,1,2}, 1 partition with 3 replicas on 0/1/2, all in sync while 0 is leader and 1 is controller, current offset is 100. And here is my hyp

Re: Problem with node after restart no partitions?

Hi, I've been through some similar problems, posted in anthor thread (I just pasted some broker logs, please check the latest message which the mail archieve

Re: Possible DEAD LOCK for one day at broker controller?

y be there are many segments that > need to be deleted all at once. Can you try may be reducing the > retention.ms > in smaller intervals - like reduce it to 9 days from 10 days and see if the > brokers are fine. > > On Tue, Aug 18, 2015 at 12:48 AM Zhao Weinan wrote: > > >

Re: Possible DEAD LOCK for one day at broker controller?

lete thread runs (every five min), it deletes > even the latest Segment because that Segment age is older than > retention.ms > > > On Mon, Aug 17, 2015 at 11:30 PM, Zhao Weinan wrote: > > > Hi Kishore Senji, > > > > The size of segement file is default 1GB. >

Re: Possible DEAD LOCK for one day at broker controller?

. > > My guess is that your segment file is huge and the OS may be taking a long > time to destage the file cache on to disk before letting it to be deleted. > This could be the reason for long pause which could be causing the Zk > connections to be timed out. > > > > On Mon

Re: Possible DEAD LOCK for one day at broker controller?

xecutor. This > could be the fix we might need. > > retention.ms config parameter should not have commas in the value. Are you > just using it here to clarify for us. > > It so happened in your > On Sun, Aug 16, 2015 at 1:52 AM Zhao Weinan wrote: > > > Hi guys,

Possible DEAD LOCK for one day at broker controller?

Hi guys, I got this problem, after changing one topic's config to retention.ms=86,400,000 from 864,000,000, the brokers start to shedule and do deletions of outdated index of that topic. Then for some reason some brokers' connection with zookeeper were expired, suddenly lots of ERRORs showed up i

Re: high level consumer api blocked forever

05 11:48 GMT+08:00 zhao weinan : > Hi, every one. > > I got into a strange case that my consumer using high level api worked > fine at first, but couple days later blocked in ConsumerIterator.hasNext(), > while there are pending messages on the topic: with > kafka-console-consumer

high level consumer api blocked forever

nyone encountered this, or it's this a known issue? Any information will be helpful. Thanks a lot! -- Zhao Weinan

Re: Is there acl or other things on kafka?

* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > / > > > On M

Is there acl or other things on kafka?

Hi, i've searched the mailing list archive, but nothing found. I'm wondering how to prevent test producer from sending dirty messages to a production broker? -- zhaown