Hi Ismael, Thank you for replying.
We are using kafka version 2.1.0 and Kafka streams version 2.0.0 Just you let you know, I was able to fix the problem by changing processing guarantee config from exactly once to at least once but now I'll need to verify if results are affected. On Sat, Feb 16, 2019 at 10:32 PM Ismael Juma <ism...@juma.me.uk> wrote: > Hi, > > What version of Kafka are you using? > > Ismael > > On Fri, Feb 15, 2019 at 8:32 PM Ankur Rana <ankur.r...@getfareye.com> > wrote: > > > Any comments anyone? > > > > On Fri, Feb 15, 2019 at 6:08 PM Ankur Rana <ankur.r...@getfareye.com> > > wrote: > > > >> Hi everyone, > >> > >> We have a Kafka cluster with 5 brokers with all topics having at least 2 > >> replication factor. We have multiple Kafka consumers applications > running > >> on this cluster. Most of these consumers are build using consumer APIs > and > >> quite recently we have started using Stream applications. > >> > >> We are facing a really weird issue. Just Sometimes it happens that our > >> Kafka cluster breaks down, By breaking down I mean that consumers and > >> producers start throwing disconnection exception and all of them just > stop. > >> > >> We use debezium connector to push Postgres events to Kafka topics. > >> Debezium throws the error below: > >> [image: image.png] > >> > >> > >> Kafka broker throws the error below: > >> COORDINATOR_NOT_AVAILABLE > >> [image: image.png] > >> > >> > >> Error on the consumer side : > >> > >> [image: image.png] > >> > >> > >> In order to fix, I stop the disconnected broker and everything fixes > >> itself. Debezium starts flushing messages and all consumers start > working > >> normally. I bring the disconnected broker up and everything works as > >> before without any problem. > >> > >> I don't understand a few things here : > >> > >> > >> 1. what could be the reason behind this disconnection exception. Even > >> if one of the broker was somehow disconnected, Isn't kafka suppose to > >> handle it in a cluster where all topics have a replication factor of > 2. > >> 2. It appears that the malfunctioning broker was in a state where it > >> was neither disconnected nor connected to the cluster. I could still > see > >> the broker visible in Kafka manager with zero bytes In, while it was > >> disconnected from all the producers and consumers. > >> 3. Weirdly, I have noticed that this situation usually occurs when I > >> start the multiple consumers of the stream application. Not sure > about this > >> as this error has only occurred a few times. It happened twice today > and > >> both the times I started 3 consumers of the same stream application. > >> > >> > >> Can anyone help me debug this problem. I don't know where to look for > >> possible issues with our cluster or stream application. I am attaching > >> streams config and stream application code for your reference. > >> Please feel free to ask for any more details. > >> > >> > >> Stream config : > >> [image: image.png] > >> > >> > >> Stream application code : https://codeshare.io/Gq6pLB > >> > >> -- > >> Thanks, > >> > >> Ankur Rana > >> Software Developer > >> FarEye > >> > > > > > > -- > > Thanks, > > > > Ankur Rana > > Software Developer > > FarEye > > > -- Thanks, Ankur Rana Software Developer FarEye
