It sounds like you have a much deeper rooted problem. Is zookeeper swapping? Something has to be causing this. After you fix this symptom you will probably start to see constant leader elections and the isr shrinking/growing and constant consumer rebalancing (or at least every minute) and a herd affect up/down stream occuring. You need to figure out what is causing the long session timeout and resolve that, IMHO. Zookeeper health is the first place to look. Next would be the network.
/******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Tue, Sep 30, 2014 at 4:57 PM, Seshadri, Balaji <balaji.sesha...@dish.com> wrote: > The zookeeper session timeout is 60 secs ,but that did not help. > > We are having broker crash and unresponsive, we got the "conditional > update" failed error when broker crashed which confirmed that it is because > of KAFKA-1382. > > server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - > Conditional update of path > /brokers/topics/dish-promo-application-access/partitions/128/state with > data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1, > "leader_epoch":99, "version":1 } and expected version 150 failed due to > org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = > BadVersion for > /brokers/topics/dish-promo-application-access/partitions/128/state > > We are in very old version 0.8-beta so it's not just patch but switching > to stable release version which also has the patch. > > -----Original Message----- > From: Joe Stein [mailto:joe.st...@stealth.ly] > Sent: Tuesday, September 30, 2014 2:01 PM > To: users@kafka.apache.org > Cc: Neha Narkhede > Subject: Re: BadVersion state in Kafka Logs > > Have you tried increasing your broker's zookeeper session timeout as a > work around for now to alleviate the issue? Is that an option for you? > Assuming that is the culprit you are timing zk sessions out and bumping into > KAFKA-1382 on the reconnect? Not knowing enough about what is going on > with the cluster it is hard to say if anything negative will come from it > but seems like it might be a an approach to try... if you can figure out > what is causing the session to timeout and fix *that* it would be a > solution also.... if it is happening every couple days (as another email > thread > states) something is going on that may not just be fixed by a single patch. > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > On Tue, Sep 30, 2014 at 11:49 AM, Seshadri, Balaji < > balaji.sesha...@dish.com > > wrote: > > > Hi Joe, > > > > I did not try on 0.8.1 branch ,I can try and see if it goes through > > when I get some breather. > > > > Thanks for initiating on 0.8.1.2. > > > > Thanks, > > > > Balaji > > > > -----Original Message----- > > From: Joe Stein [mailto:joe.st...@stealth.ly] > > Sent: Tuesday, September 30, 2014 9:34 AM > > To: users@kafka.apache.org > > Cc: Neha Narkhede > > Subject: Re: BadVersion state in Kafka Logs > > > > Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if you > > could make a patch that does would be great. > > > > I will kick off a discussion for KAFKA-1382 and the scala 2.11 for > > 0.8.1.2 release (and see what others may think we should do like the > > gradle changes I think we should do too for src release issues (and > > the jars in the repo)). I will send that on dev/user in a little bit > > (please comment +1 community support please on that thread for the > release). > > > > /******************************************* > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > ********************************************/ > > > > On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji < > > balaji.sesha...@dish.com > > > wrote: > > > > > I would love to help you guys to make Kafka best in Pub/Sub, will > > > continue doing that whenever I can. > > > > > > Do we have 0.8.1.2 release tag or should we apply patch on top of > > > 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?. > > > > > > Balaji > > > > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > > Sent: Monday, September 29, 2014 5:21 PM > > > To: Seshadri, Balaji > > > Cc: users@kafka.apache.org > > > Subject: Re: BadVersion state in Kafka Logs > > > > > > It is difficult to predict an exact date. Though all the discussions > > > of the progress and ETA are on the mailing list. You can follow the > > > discussions to know the details and/or offer to help out on the > > > outstanding issues. > > > > > > On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji < > > > balaji.sesha...@dish.com<mailto:balaji.sesha...@dish.com>> wrote: > > > Neha, > > > > > > Do you know the date in Oct when 0.8.2 is going to be out ?. > > > > > > Thanks, > > > > > > Balaji > > > > > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com<mailto: > > > neha.narkh...@gmail.com>] > > > Sent: Thursday, September 25, 2014 1:08 PM > > > To: Seshadri, Balaji > > > Cc: users@kafka.apache.org<mailto:users@kafka.apache.org> > > > > > > Subject: Re: BadVersion state in Kafka Logs > > > > > > We are close to the release. I'd probably expect 0.8.2 sometime in > > October. > > > > > > On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji < > > > balaji.sesha...@dish.com<mailto:balaji.sesha...@dish.com>> wrote: > > > Hi Neha, > > > > > > Do you know when are you guys releasing 0.8.2 ?. > > > > > > Thanks, > > > > > > Balaji > > > > > > -----Original Message----- > > > From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com<mailto: > > > balaji.sesha...@dish.com>] > > > Sent: Thursday, September 25, 2014 9:41 AM > > > To: users@kafka.apache.org<mailto:users@kafka.apache.org> > > > Subject: RE: BadVersion state in Kafka Logs > > > > > > Thanks for the replay. > > > > > > Please let me know if we can use trunk as 0.8.2 is not yet released. > > > > > > Balaji > > > ________________________________________ > > > From: Neha Narkhede [neha.narkh...@gmail.com<mailto: > > > neha.narkh...@gmail.com>] > > > Sent: Wednesday, September 24, 2014 6:32 PM > > > To: users@kafka.apache.org<mailto:users@kafka.apache.org> > > > Subject: Re: BadVersion state in Kafka Logs > > > > > > From the logs you've attached, my guess is it's most likely due to > > > KAFKA-1382. > > > > > > Thanks, > > > Neha > > > > > > On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji < > > > balaji.sesha...@dish.com<mailto:balaji.sesha...@dish.com> > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > > > We got the below error in our logs and our consumers stopped > > > > consuming any data ?.It worked only after restart. > > > > > > > > > > > > > > > > We would like to confirm that it's because we are running with > > > > 0.8-beta version and not 0.8 release version to convince "THE MGMT" > > guys. > > > > > > > > > > > > > > > > Please let me know if it's this KAFKA-1382 causing the issue. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Balaji > > > > > > > > > > > > > > > > *From:* Gulia, Vikram > > > > *Sent:* Wednesday, September 24, 2014 8:43 AM > > > > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA > > > > *Cc:* Alam, Mohammad Shah > > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions. > > > > > > > > > > > > > > > > Adding full MAA distro. > > > > > > > > > > > > > > > > DES Offshore looked in to the logs on kafka servers and seems like > > > > the issue we encountered yesterday may be described in these > > > > threads, please have a look - > > > > > > > > > > > > > > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904 > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the > > > > fix/patch which is available in > > > > 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>) > > > > > > > > > > > > > > > > Thank You, > > > > > > > > Vikram Gulia > > > > > > > > > > > > > > > > *From:* Sharma, Navdeep > > > > *Sent:* Wednesday, September 24, 2014 6:53 AM > > > > *To:* Gulia, Vikram; #IT-MAD DES > > > > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah > > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions. > > > > > > > > > > > > > > > > Hi Vikram, > > > > > > > > > > > > > > > > We analyzed below mentioned issue with MAA-Offshore (Abhishek) > > > > and found that the error occurred only on 23 Sept. This is not > > > > historical as we checked last 4 days logs. > > > > > > > > > > > > > > > > It looks like that consumer got stopped on September 22 2014 for > > > > Linux patching activty.MAA started consumer September 23 2014 at > > > > 1:00 > > AM. > > > > > > > > > > > > > > > > *Issue *in server log *"* *BadVersion for > > > > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not > > > > present in previous 4 days logs. > > > > > > > > More detail of this error can be found at- > > > > > > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904 > > > > > > > > > > > > > > > > We are not sure about data loss in this scenario and working on this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let us know if any concerns. > > > > > > > > > > > > > > > > [image: cid:image001.gif@01CF7B0A.03F21580] > > > > > > > > Navdeep Sharma > > > > Developer - offshore, Middleware Applications & Development o India: > > > > 0120-4532000 - 2234 > > > > c: +91-9911698102<tel:%2B91-9911698102> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *From:* Gulia, Vikram > > > > *Sent:* Tuesday, September 23, 2014 6:17 PM > > > > *To:* #IT-MAD DES > > > > *Subject:* FW: 9/23 prod issue - offline kafka partitions. > > > > > > > > > > > > > > > > DES Offshore dev, > > > > > > > > > > > > > > > > Please work with MAA offshore to monitor the kafka broker as we > > > > had this incident where lot of partitions went offline around 1.45 > > > > PM MST and MAA has to restart the kafka servers. We may have lost > > > > messages and we need to see if there is a way to figure out what > > > > was > > the impact. > > > > > > > > > > > > > > > > Also, check the logs for kafka servers and see if we can figure > > > > out why did partitions go offline or are un-available? Let us know > > > > if you find anything relevant. > > > > > > > > > > > > > > > > Thank You, > > > > > > > > Vikram Gulia > > > > > > > > > > > > > > > > *From:* Nielsen, Andy > > > > *Sent:* Tuesday, September 23, 2014 5:04 PM > > > > *To:* #IT-MAD DES; Gulia, Vikram > > > > *Cc:* #IT-MAA > > > > *Subject:* 9/23 prod issue - offline kafka partitions. > > > > > > > > > > > > > > > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic > > > > --unavailable-partitions > > > > > > > > topic: account-access partition: 21 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: account-access partition: 51 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: account-access partition: 81 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: account-access partition: 111 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: account-activated partition: 13 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: account-activated partition: 43 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: account-activated partition: 73 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: account-activated partition: 103 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: account-adjustment-issued partition: 27 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: account-adjustment-issued partition: 57 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: account-adjustment-issued partition: 87 leader: -1 > > > > replicas: 4,6,1 isr: 1 > > > > > > > > topic: account-adjustment-issued partition: 117 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: account-created partition: 11 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: account-created partition: 41 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: account-created partition: 71 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: account-created partition: 101 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: account-info-updated partition: 7 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: account-info-updated partition: 37 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: account-info-updated partition: 67 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: account-info-updated partition: 97 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: account-info-updated partition: 127 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: application-access partition: 21 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: application-access partition: 51 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: application-access partition: 81 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: application-access partition: 111 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: bill-generated partition: 3 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: bill-generated partition: 33 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: bill-generated partition: 63 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: bill-generated partition: 93 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: bill-generated partition: 123 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: collected-event partition: 29 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: collected-event partition: 59 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: collected-event partition: 89 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: collected-event partition: 119 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: customer-cues partition: 27 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: customer-cues partition: 57 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: customer-cues partition: 87 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: customer-cues partition: 117 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: dish-promo-application-access partition: 23 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: dish-promo-application-access partition: 53 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: dish-promo-application-access partition: 83 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: dish-promo-application-access partition: 113 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: event-response partition: 2 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: event-response partition: 32 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: event-response partition: 62 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: event-response partition: 92 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: event-response partition: 122 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: leads-service partition: 24 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: leads-service partition: 54 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: leads-service partition: 84 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: leads-service partition: 114 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: logprod_v3 partition: 3 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: logprod_v3 partition: 33 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: logprod_v3 partition: 63 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: logprod_v3 partition: 93 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: logprod_v3 partition: 123 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: online-account-registration-attempted partition: 21 > leader: > > > > -1 replicas: 4,6,1 isr: > > > > > > > > topic: online-account-registration-attempted partition: 51 > leader: > > > > -1 replicas: 4,6,1 isr: 1 > > > > > > > > topic: online-account-registration-attempted partition: 81 > leader: > > > > -1 replicas: 4,6,1 isr: > > > > > > > > topic: online-account-registration-attempted partition: 111 > leader: > > > > -1 replicas: 4,6,1 isr: > > > > > > > > topic: order-cancelled partition: 29 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-cancelled partition: 59 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-cancelled partition: 89 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-cancelled partition: 119 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: order-completed partition: 24 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-completed partition: 54 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-completed partition: 84 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: order-completed partition: 114 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-created partition: 25 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-created partition: 55 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-created partition: 85 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-created partition: 115 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-modified partition: 8 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: order-modified partition: 38 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-modified partition: 68 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-modified partition: 98 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-modified partition: 128 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: order-request partition: 24 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-request partition: 54 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-request partition: 84 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: order-request partition: 114 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-response partition: 27 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: order-response partition: 57 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-response partition: 87 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: order-response partition: 117 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: outbound-call-attempted partition: 13 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: outbound-call-attempted partition: 43 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: outbound-call-attempted partition: 73 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: outbound-call-attempted partition: 103 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: outbound-communications partition: 4 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: outbound-communications partition: 34 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: outbound-communications partition: 64 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: outbound-communications partition: 94 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: outbound-communications partition: 124 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > topic: postal-mail-undeliverable partition: 15 leader: -1 > > > > replicas: 4,6,1 isr: 1 > > > > > > > > topic: postal-mail-undeliverable partition: 45 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: postal-mail-undeliverable partition: 75 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: postal-mail-undeliverable partition: 105 leader: -1 > > > > replicas: 4,6,1 isr: > > > > > > > > topic: rain-burn-in partition: 4 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-burn-in partition: 34 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-burn-in partition: 64 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-burn-in partition: 94 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-burn-in partition: 124 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-enhanced partition: 26 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-enhanced partition: 56 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-enhanced partition: 86 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-enhanced partition: 116 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-listener partition: 23 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-listener partition: 53 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-listener partition: 83 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-listener partition: 113 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-load-test partition: 8 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-load-test partition: 38 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-load-test partition: 68 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: rain-load-test partition: 98 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: rain-load-test partition: 128 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: submit-agreement partition: 2 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: submit-agreement partition: 32 leader: -1 replicas: > 4,6,1 > > > > isr: 1 > > > > > > > > topic: submit-agreement partition: 62 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: submit-agreement partition: 92 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: submit-agreement partition: 122 leader: -1 replicas: > 4,6,1 > > > > isr: > > > > > > > > topic: threshold-exceeded partition: 14 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: threshold-exceeded partition: 44 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: threshold-exceeded partition: 74 leader: -1 > > replicas: > > > > 4,6,1 isr: > > > > > > > > topic: threshold-exceeded partition: 104 leader: -1 > > replicas: > > > > 4,6,1 isr: 1 > > > > > > > > > > > > > > > > *Andy Nielsen* > > > > > > > > *Middleware Application Admin* > > > > > > > > *303-723-2347<tel:303-723-2347> <303-723-2347<tel:303-723-2347>>* > > > > > > > > *cell:720-971-2856<tel:720-971-2856> > > > > <720-971-2856<tel:720-971-2856>>* > > > > > > > > > > > > > > > > > > > > > > > >
