Re: All brokers are running but some partitions' leader is -1

2015-11-25 Thread Qi Xu
Great to know that. Thanks Gwen! On Wed, Nov 25, 2015 at 12:03 PM, Gwen Shapira wrote: > 1. Yes, you can do a rolling upgrade of brokers from 0.8.2 to 0.9.0. The > important thing is to upgrade the brokers before you upgrade any of the > clients. > > 2. I'm not aware of issues with 0.9.0 and Spa

Re: All brokers are running but some partitions' leader is -1

2015-11-25 Thread Gwen Shapira
1. Yes, you can do a rolling upgrade of brokers from 0.8.2 to 0.9.0. The important thing is to upgrade the brokers before you upgrade any of the clients. 2. I'm not aware of issues with 0.9.0 and SparkStreaming. However, definitely do your own testing to make sure. On Wed, Nov 25, 2015 at 11:25 A

Re: All brokers are running but some partitions' leader is -1

2015-11-25 Thread Qi Xu
Hi Gwen, Yes, we're going to upgrade the 0.9.0 version. Regarding the upgrade, we definitely don't want to have down time of our cluster. So the upgrade will be machine by machine. Will the release 0.9.0 work with the Aug's version together in the same Kafka cluster? Also we currently run spark str

Re: All brokers are running but some partitions' leader is -1

2015-11-24 Thread Qi Xu
Thanks a lot Prabhjot! The issue is mitigated by running the preferred replica leader election tool! Before that, I noticed that it simply could not do leader election---when I created a new topic, that topic is not available for a long time until preferred replica leader election finishes. For th

Re: All brokers are running but some partitions' leader is -1

2015-11-23 Thread Prabhjot Bharaj
Hi, With the information provided, these are the steps I can think of (based on the experience I had with kafka):- 1. do a describe on the topic. See if the partitions and replicas are evenly distributed amongst all. If not, you might want to try the 'Reassign Partitions Tool' - https://cwiki.apa

Re: All brokers are running but some partitions' leader is -1

2015-11-23 Thread Gwen Shapira
We fixed many many bugs since August. Since we are about to release 0.9.0 (with SSL!), maybe wait a day and go with a released and tested version. On Mon, Nov 23, 2015 at 3:01 PM, Qi Xu wrote: > Forgot to mention is that the Kafka version we're using is from Aug's > Trunk branch---which has the

Re: All brokers are running but some partitions' leader is -1

2015-11-23 Thread Qi Xu
Forgot to mention is that the Kafka version we're using is from Aug's Trunk branch---which has the SSL support. Thanks again, Qi On Mon, Nov 23, 2015 at 2:29 PM, Qi Xu wrote: > Loop another guy from our team. > > On Mon, Nov 23, 2015 at 2:26 PM, Qi Xu wrote: > >> Hi folks, >> We have a 10 nod

Re: All brokers are running but some partitions' leader is -1

2015-11-23 Thread Qi Xu
Loop another guy from our team. On Mon, Nov 23, 2015 at 2:26 PM, Qi Xu wrote: > Hi folks, > We have a 10 node cluster and have several topics. Each topic has about > 256 partitions with 3 replica factor. Now we run into an issue that in some > topic, a few partition (< 10)'s leader is -1 and all

All brokers are running but some partitions' leader is -1

2015-11-23 Thread Qi Xu
Hi folks, We have a 10 node cluster and have several topics. Each topic has about 256 partitions with 3 replica factor. Now we run into an issue that in some topic, a few partition (< 10)'s leader is -1 and all of them has only one synced partition. >From the Kafka manager, here's the snapshot: [i