Network failure leads to split brain in brokers

Hi Guys, We have a 6 node cluster. Due to network failure yesterday(network was up and running in few minutes), few of the brokers were not able to talk to each other, including the zookeeper. During an issue in our camus job, we identified that broker-2 was misbehaving. The controller logs reflec

Re: ISR not updating

Yes. it contain my server.properties file. auto.leader.rebalance.enable=true On Wed, Jun 18, 2014 at 12:44 PM, Jun Rao wrote: > Did you have auto leader balancing enabled? > > Thanks, > > Jun > > > On Tue, Jun 17, 2014 at 5:06 PM, Bongyeon Kim > wrote: > > > There is some error log about fai

SimpleConsumer kafka.api.OffsetRequest.LatestTime() not working ?

Simple Consumer is *not processing* messages when it's offset in kafka.api.OffsetRequest.*LatestTime()* . If i use *EarlierTime() *offset method it is processing from start, which is as expected. But when I try for *LatestTime()* offset, no reaction from simple consumer and I am referring below

Kafka latency measures

Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are

delete topic ?

Hi, Doing some evaluation testing, and accidently create a queue with wrong replication factor. Trying to delete as in: kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic replicated-topic Yeilded: Command must include exactly one action: --list, --describe, --cr

Re: MirrorMaker documentation suggestions

I just granted you the wiki permission. Could you give it a try? Thanks, Jun On Tue, Jun 17, 2014 at 11:30 PM, Daniel Compton wrote: > Hi Guozhang > > I feel pretty silly for asking this but how do I edit the Confluence Wiki? > I have an account and it says there is no edit restrictions defin

Re: ISR not updating

Auto leader balancing has some know issues when using together with controlled shutdown. So, we don't recommend it to be turned on in 0.8.1.1 Thanks, Jun On Wed, Jun 18, 2014 at 1:41 AM, Bongyeon Kim wrote: > Yes. > it contain my server.properties file. > > > auto.leader.rebalance.enable=true

Re: Kafka latency measures

Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva : > Hi, > > We are trying to evaluate Kafka for a real time application. We are sen

Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

Hi folks, In my project, we want to perform to update our active Kafka 0.8 cluster to Kafka 0.8.1.1 without downtime and losing any data. The process (after reading http://kafka.apache.org/documentation.html#upgrade) looks to me like this. For each broker in turn: 1. Bring the broker down. 2. Upd

Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

You don't gain much by running #4 between broker bounces. Running it after the cluster is upgraded will be sufficient. Thanks, Neha On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin wrote: > Hi folks, > > In my project, we want to perform to update our active Kafka 0.8 cluster to > Kafka 0.8.1.1 wi

Re: Kafka latency measures

Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill wrote: > Hi, > > do these spikes happen to correlate

Re: SimpleConsumer kafka.api.OffsetRequest.LatestTime() not working ?

Is that because you have no more data produced to the topic/partition after the last offset returned by OffsetRequest.LatestTime() ? On Wed, Jun 18, 2014 at 6:19 AM, rafeeq s wrote: > Simple Consumer is *not processing* messages when it's offset in > kafka.api.OffsetRequest.*LatestTime()* . > >

Re: delete topic ?

Kafka allows increasing the replication factor of a topic. You can read about it here . We do not support reducing the number of partitions, so you either have to create a new topic or delete the existing one. We

Re: delete topic ?

When we were in testing phase, we would either create a new topic with the correct details or shut the cluster down and hard kill the topic in zookeeper + local disk. In prod we have the cluster configured via configuration management and auto create turned off. The ability to delete a topic i

Re: Building Kafka on Mac OS X

I will let you know once I come back from my job. Jorge On mié 18 jun 2014 00:35:02 AST, Timothy Chen wrote: So do you have the build folder generated in core/client and classes/jars compiled in them? Kafka-server-start.sh also sets the CLASSPATH to load the jar and kafka.Kafka as well, so you

Re: Kafka latency measures

The spikes happens without any correlation with the log.flush.interval.message. They happen more frequently. I'm using the latest version. I'm sending the messages to Kafka, then there is a message receiver, it sends the same messages back through kafka to original sender. The round trip latency i

Re: SimpleConsumer kafka.api.OffsetRequest.LatestTime() not working ?

Hi Refeeq, I'm using the LatestTime() and it is working. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:07 PM, Neha Narkhede wrote: > Is that because you have no more data produced to the topic/partition after > the last offset returned by OffsetRequest.LatestTime() ? > > > On Wed, Jun 18, 2014 a

Re: Kafka latency measures

I've found this performance test. http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ This performance test has mentioned about the same issue at the end. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva wrote: > The spikes happens without any correlati

Running mirrormaker in HA?

Hi, I've looked at the documentation for mirrormaker. It doesnt appear that there is a notion of HA ( high availability ) for the mirrormaker tool itself. Ideally, I would like to have some topics flowing to another datacenter with a 2 node mirrormaker in between, where one mirrormaker is always a

Re: Network failure leads to split brain in brokers

Hello Abhinav, Which Kafka version are you using? Guozhang On Wed, Jun 18, 2014 at 1:40 AM, Abhinav Anand wrote: > Hi Guys, > We have a 6 node cluster. Due to network failure yesterday(network was up > and running in few minutes), few of the brokers were not able to talk to > each other, inc

Re: delete topic ?

I'm using 0.8.1.1 I use DeleteTopicCommand to delete topic args[0] = "--topic"; args[1] = the topic you want to delete args[2] = "--zookeeper"; args[3] = kafkaZookeepers; DeleteTopicCommand.main(args); You can write your own script to delete the topic, I guess. And I think it o

Re: Running mirrormaker in HA?

Hi Prakash, Actually the MM is HA such that it used a consumer group fetching data from the source DC and another producer pushing them to the destination DC. So suppose there are two instances of MM as in your case, each of the instance will get half of the traffic mirroring to the other DC since

Re: delete topic ?

Yes the existing delete topic command just cleans up the topic entry in zk, but not really deleting the topic from the cluster. I have a patch that enables kafka-topics.sh to delete topic but not sure if it's merged to trunk. Tim > On Jun 18, 2014, at 1:39 PM, "hsy...@gmail.com" wrote: > > I

Re: Running mirrormaker in HA?

I see what you mean by the implicit HA.Thanks Guozhang. On Wed, Jun 18, 2014 at 1:45 PM, Guozhang Wang wrote: > Hi Prakash, > > Actually the MM is HA such that it used a consumer group fetching data from > the source DC and another producer pushing them to the destination DC. So > suppose ther

Re: Reliable Message Commits

I think I've discovered the answer to my second question: according to the code in ZookeeperConsumerConnector.scala, a rebalance derives its offsets from what's already in Zookeeper. Therefore, uncommitted but consumed messages from a given partition will be replayed when the partition is reassigne

Re: MirrorMaker documentation suggestions

Hi Jun Thanks, I was able to update the wiki. Daniel. On Thursday, 19 June 2014 at 2:45 am, Jun Rao wrote: > I just granted you the wiki permission. Could you give it a try? > > Thanks, > > Jun > > > On Tue, Jun 17, 2014 at 11:30 PM, Daniel Compton (mailto:d...@danielcompton.net)> >

Re: Building Kafka on Mac OS X

Everything ran smooth now, I will start running tests. Thanks. Jorge On Jun 18, 2014, at 11:44 AM, Jorge Marizan wrote: > I will let you know once I come back from my job. > > Jorge > > On mié 18 jun 2014 00:35:02 AST, Timothy Chen wrote: >> So do you have the build folder generated in core/c

Re: ISR not updating

I'm not enable controllerd.shutdown property. anyway, I'd better disable auto leader banancing. I would like to find that information on documentation. Thanks. Jun On Wed, Jun 18, 2014 at 11:47 PM, Jun Rao wrote: > Auto leader balancing has some know issues when using together with > control

Re: Kafka latency measures

what are the latency numbers you observed, avg as well as worst case? Here is a blog that we did recently which should reflect latest performance metrics for latency - http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines On Wed, Jun 18, 2014

Re: Kafka latency measures

My machine configuration is not very high. The average one way latency we observe is around 10 ~ 15 ms for 50k messages. The outliers doesn't occur for small messages. For small messages we observe around 6 ms latency. Thanks, Supun.. On Wed, Jun 18, 2014 at 10:18 PM, Neha Narkhede wrote: > wh

Re: Reliable Message Commits

Hi Kyle, Thanks for the update. Wondering if you found answer to your N-1 commit question? If auto commit happens only at iterator.next () and onky for the N -1 message then client code can be much simpler and reliable as you mentioned. I'm also looking forward to any post in this regard. J

Re: SimpleConsumer kafka.api.OffsetRequest.LatestTime() not working ?

Thanks to all. LatestTime() is working as expected. Regards, Rafeeq S *(“What you do is what matters, not what you think or say or plan.” )* On Wed, Jun 18, 2014 at 8:40 PM, Supun Kamburugamuva wrote: > Hi Refeeq, > > I'm using the LatestTime() and it is working. > > Thanks, > Supun.. > >