Re: Broker suddenly becomes unstable after upgrade to 2.1.0

2019-03-18 Thread Joe Ammann
Hi all On 3/13/19 9:36 AM, Joe Ammann wrote: > Hi Ankur > > On 3/13/19 3:34 AM, Ankur Rana wrote: >> Hey, >> I think this is a known issue in Kafka 2.1.0. Check this out >> https://issues.apache.org/jira/browse/KAFKA-7697 >> It has been fixed in 2.1.1. > This surely does look like our issue! I sho

Re: Broker suddenly becomes unstable after upgrade to 2.1.0

2019-03-13 Thread Joe Ammann
Hi Ankur On 3/13/19 3:34 AM, Ankur Rana wrote: > Hey, > I think this is a known issue in Kafka 2.1.0. Check this out > https://issues.apache.org/jira/browse/KAFKA-7697 > It has been fixed in 2.1.1. This surely does look like our issue! I should have found that myself.. Thanks, we'll roll out 2.1

Re: Broker suddenly becomes unstable after upgrade to 2.1.0

2019-03-12 Thread Ankur Rana
Hey, I think this is a known issue in Kafka 2.1.0. Check this out https://issues.apache.org/jira/browse/KAFKA-7697 It has been fixed in 2.1.1. On Wed, Mar 13, 2019 at 12:25 AM Joe Ammann wrote: > Yes, to the best of our knowledge. We had the option to take the cluster > down on Sat, so we stopp

Re: Broker suddenly becomes unstable after upgrade to 2.1.0

2019-03-12 Thread Joe Ammann
Yes, to the best of our knowledge. We had the option to take the cluster down on Sat, so we stopped everything, upgraded the software, set interbroker and message version to 2.1 and restarted. On Sun we did another rolling restart of all 4 brokers. All looked good after that (and we had done th

Re: Broker suddenly becomes unstable after upgrade to 2.1.0

2019-03-12 Thread M. Manna
When you upgraded, did you follow the rolling upgrade instruction from Kafka site ? And changed message version etc. as described on the site ? Thanks, On Tue, 12 Mar 2019 at 23:20, Joe Ammann wrote: > Hi all > > last weekend we have upgraded a small cluster (4 brokers) from 0.10.2.1 > to 2.1.

Broker suddenly becomes unstable after upgrade to 2.1.0

2019-03-12 Thread Joe Ammann
Hi all last weekend we have upgraded a small cluster (4 brokers) from 0.10.2.1 to 2.1.0. Everything seemed to go well on the weekend. For 2 nights in a row we now had an strange behaviour on one of the 4 nodes. At almost exactly 00:08 in both nights, 1 out of the 4 brokers stopped writing anythin