Hi Everyone, Has anyone ever faced too many CLOSE_WAIT connections in Kafka broker? We are facing the same problem in our Kafka cluster.
tcp6 27 0 172.31.10.143:9092 172.31.0.47:45138 ESTABLISHED - tcp6 25 0 172.31.10.143:9092 172.31.46.69:41612 CLOSE_WAIT - tcp6 25 0 172.31.10.143:9092 172.31.0.47:45010 CLOSE_WAIT - tcp6 25 0 172.31.10.143:9092 172.31.46.69:43000 CLOSE_WAIT - tcp6 194 0 172.31.10.143:8080 172.31.20.219:45952 CLOSE_WAIT - tcp6 25 0 172.31.10.143:9092 172.31.20.219:48006 CLOSE_WAIT - tcp6 1 0 172.31.10.143:9092 172.31.0.47:44582 CLOSE_WAIT - tcp6 25 0 172.31.10.143:9092 172.31.46.69:42828 CLOSE_WAIT - ... 1000s of such connections in close wait state causing DisconnectException in Kafka Clients. We are using Kafka 2.1.0. Are there any known issues around the same? Is it possible that the issue is with Kafka consumer causing this problem? Any ideas? On Sun, Feb 17, 2019 at 3:30 AM Ankur Rana <ankur.r...@getfareye.com> wrote: > Hi Ismael, > > Thank you for replying. > > We are using kafka version 2.1.0 > and Kafka streams version 2.0.0 > > Just you let you know, I was able to fix the problem by changing > processing guarantee config from exactly once to at least once but now I'll > need to verify if results are affected. > > > > > > > On Sat, Feb 16, 2019 at 10:32 PM Ismael Juma <ism...@juma.me.uk> wrote: > >> Hi, >> >> What version of Kafka are you using? >> >> Ismael >> >> On Fri, Feb 15, 2019 at 8:32 PM Ankur Rana <ankur.r...@getfareye.com> >> wrote: >> >> > Any comments anyone? >> > >> > On Fri, Feb 15, 2019 at 6:08 PM Ankur Rana <ankur.r...@getfareye.com> >> > wrote: >> > >> >> Hi everyone, >> >> >> >> We have a Kafka cluster with 5 brokers with all topics having at least >> 2 >> >> replication factor. We have multiple Kafka consumers applications >> running >> >> on this cluster. Most of these consumers are build using consumer APIs >> and >> >> quite recently we have started using Stream applications. >> >> >> >> We are facing a really weird issue. Just Sometimes it happens that our >> >> Kafka cluster breaks down, By breaking down I mean that consumers and >> >> producers start throwing disconnection exception and all of them just >> stop. >> >> >> >> We use debezium connector to push Postgres events to Kafka topics. >> >> Debezium throws the error below: >> >> [image: image.png] >> >> >> >> >> >> Kafka broker throws the error below: >> >> COORDINATOR_NOT_AVAILABLE >> >> [image: image.png] >> >> >> >> >> >> Error on the consumer side : >> >> >> >> [image: image.png] >> >> >> >> >> >> In order to fix, I stop the disconnected broker and everything fixes >> >> itself. Debezium starts flushing messages and all consumers start >> working >> >> normally. I bring the disconnected broker up and everything works as >> >> before without any problem. >> >> >> >> I don't understand a few things here : >> >> >> >> >> >> 1. what could be the reason behind this disconnection exception. >> Even >> >> if one of the broker was somehow disconnected, Isn't kafka suppose >> to >> >> handle it in a cluster where all topics have a replication factor >> of 2. >> >> 2. It appears that the malfunctioning broker was in a state where it >> >> was neither disconnected nor connected to the cluster. I could >> still see >> >> the broker visible in Kafka manager with zero bytes In, while it was >> >> disconnected from all the producers and consumers. >> >> 3. Weirdly, I have noticed that this situation usually occurs when I >> >> start the multiple consumers of the stream application. Not sure >> about this >> >> as this error has only occurred a few times. It happened twice >> today and >> >> both the times I started 3 consumers of the same stream application. >> >> >> >> >> >> Can anyone help me debug this problem. I don't know where to look for >> >> possible issues with our cluster or stream application. I am attaching >> >> streams config and stream application code for your reference. >> >> Please feel free to ask for any more details. >> >> >> >> >> >> Stream config : >> >> [image: image.png] >> >> >> >> >> >> Stream application code : https://codeshare.io/Gq6pLB >> >> >> >> -- >> >> Thanks, >> >> >> >> Ankur Rana >> >> Software Developer >> >> FarEye >> >> >> > >> > >> > -- >> > Thanks, >> > >> > Ankur Rana >> > Software Developer >> > FarEye >> > >> > > > -- > Thanks, > > Ankur Rana > Software Developer > FarEye > -- Thanks, Ankur Rana Software Developer FarEye
