Hello, Can someone please help me to understand what is the issue ?
Regards, Praveen Kumar K S +91-9986855625 On Thu, Oct 22, 2020 at 6:52 AM Praveen Kumar K S <prav...@securelyshare.com> wrote: > Hello Experts, > > Any help to debug and resolve this issue is highly appreciated. > > Regards, > Praveen > > On Wed, 21 Oct, 2020, 11:26 Sabina Marx, <sabina.m...@sneo.io> wrote: > >> Hi Praveen, >> >> it seems to be the same problem, your log looks quite similar to mine. >> But I have no solution until now. >> >> Regards >> Sabina >> >> Von: Praveen Kumar K S <prav...@securelyshare.com> >> Antworten an: "users@kafka.apache.org" <users@kafka.apache.org> >> Datum: Dienstag, 20. Oktober 2020 um 20:07 >> An: "users@kafka.apache.org" <users@kafka.apache.org> >> Betreff: Re: Client session timed out >> >> Hello, >> >> I'm not sure if I can add my issue in this thread. But it seems like I'm >> facing the same problem. >> >> KAFKA_VERSION=2.5.1 >> ZK_VERSION=3.5.8 >> >> I run 3 node zookeeper cluster and 3 node kafka cluster as docker >> containers in docker swarm environment. When I install it for first time, >> everything goes well. Zookeeper and Kafka are able to form the cluster. >> Services are healthy. >> >> But when I issue docker update command, kafka is not coming up though the >> zookeeper cluster is healthy. Below is the sequence of steps. >> >> docker service update one_zookeeper --image x.x.x/v1/zookeeper:latest >> --force >> docker service update one_zookeeper1 --image x.x.x/v1/zookeeper:latest >> --force >> docker service update one_zookeeper2 --image x.x.x/v1/zookeeper:latest >> --force >> >> Zookeeper is healthy now. I'm able to query leader and follower. >> >> Now, I'm updating kafka and it doesn't work. >> docker service update one_kafka --image x.x.com/v1/kafka:latest< >> http://x.x.com/v1/kafka:latest> --force >> >> PFA Kafka log. >> >> While kafka update has failed, I see that kafka1 and kafka2 are running >> and healthy. >> >> docker service ls | grep kafka >> one_kafka replicated 0/1 >> one_kafka1 replicated 1/1 >> one_kafka2 replicated 1/1 >> >> To cross verify, I have just brought down the services zookeeper and >> kafka without data loss. I preserve >> zookeeperdata,zookeeperlogs,zookeepertxns and kafkadata,kafkalogs. >> >> docker stack remove one >> docker stack deploy -c cluster-zookeeper.yml one >> docker stack deploy -c cluster-kafka.yml one >> >> Now, all the services are healthy. >> >> I'm not sure why kafka deployment is failing only during update. There is >> no change in the configuration in either zookeeper or kafka. >> >> Please help me resolve this issue and let me know if you need any >> additional details. >> >> Regards, >> Praveen Kumar K S >> +91-9986855625 >> >> >> On Tue, Oct 20, 2020 at 3:54 PM Sabina Marx <sabina.m...@sneo.io<mailto: >> sabina.m...@sneo.io>> wrote: >> Yes, it's the same problem. >> >> Am 19.10.20, 19:50 schrieb "Mich Talebzadeh" <mich.talebza...@gmail.com >> <mailto:mich.talebza...@gmail.com>>: >> >> can you try to disable automatic start and on the node just booted, >> start >> zookeeper first, check the log that it is connected and then start >> Kafka? >> >> I assume everything is set-up OK including in >> $KAFKA_HOME/config/server<N>.properties values for broker.id< >> http://broker.id>, hostname, >> zookeeper.connect=<server1>:2181,,server2>:2181, <serverN>:2181 and >> also >> zookeeper.connection.timeout.ms< >> http://zookeeper.connection.timeout.ms>=6000 (default) >> >> HTH >> >> >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> < >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >* >> >> >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any >> loss, damage or destruction of data or any other property which may >> arise >> from relying on this email's technical content is explicitly >> disclaimed. >> The author will in no case be liable for any monetary damages arising >> from >> such loss, damage or destruction. >> >> >> >> >> On Mon, 19 Oct 2020 at 18:17, Sabina Marx <sabina.m...@sneo.io >> <mailto:sabina.m...@sneo.io>> wrote: >> >> > Yes, you have it >> > >> > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef> >> > ________________________________ >> > Von: Mich Talebzadeh <mich.talebza...@gmail.com<mailto: >> mich.talebza...@gmail.com>> >> > Gesendet: Monday, October 19, 2020 7:09:53 PM >> > An: users@kafka.apache.org<mailto:users@kafka.apache.org> < >> users@kafka.apache.org<mailto:users@kafka.apache.org>> >> > Betreff: Re: Client session timed out >> > >> > Ok I think it is clearer now. >> > >> > As I understand all your Zookeepers and Kafkas are running. (5 nodes >> > meaning 5 physical hosts?). Then you have to reboot one physical >> host. You >> > still have the redundancy. But when the physical host comes up and >> your >> > zookeeper and then Kafka come up, you have Kafka timing out and not >> > connecting to the existing Kafka cluster? >> > >> > Does that make sense? >> > >> > >> > >> > >> > >> > >> > LinkedIn * >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > < >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > >* >> > >> > >> > >> > >> > >> > *Disclaimer:* Use it at your own risk. Any and all responsibility >> for any >> > loss, damage or destruction of data or any other property which may >> arise >> > from relying on this email's technical content is explicitly >> disclaimed. >> > The author will in no case be liable for any monetary damages >> arising from >> > such loss, damage or destruction. >> > >> > >> > >> > >> > On Mon, 19 Oct 2020 at 17:59, Sabina Marx <sabina.m...@sneo.io >> <mailto:sabina.m...@sneo.io>> wrote: >> > >> > > No, sorry I‘m not so good in explaining. >> > > The scenario is: the complete Cluster is running, all zookeepers >> and all >> > > kafkas. And then I restart one server, the others are still >> running. >> > > >> > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef> >> > > ________________________________ >> > > Von: Mich Talebzadeh <mich.talebza...@gmail.com<mailto: >> mich.talebza...@gmail.com>> >> > > Gesendet: Monday, October 19, 2020 6:46:49 PM >> > > An: users@kafka.apache.org<mailto:users@kafka.apache.org> < >> users@kafka.apache.org<mailto:users@kafka.apache.org>> >> > > Betreff: Re: Client session timed out >> > > >> > > Can you pls clarify when you say you start one pair (Zookeeper and >> > Kafka), >> > > what happens to the others. Do you keep them down? >> > > >> > > >> > > >> > > >> > > LinkedIn * >> > > >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > > < >> > > >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > > >* >> > > >> > > >> > > >> > > >> > > >> > > *Disclaimer:* Use it at your own risk. Any and all responsibility >> for any >> > > loss, damage or destruction of data or any other property which >> may arise >> > > from relying on this email's technical content is explicitly >> disclaimed. >> > > The author will in no case be liable for any monetary damages >> arising >> > from >> > > such loss, damage or destruction. >> > > >> > > >> > > >> > > >> > > On Mon, 19 Oct 2020 at 17:33, Sabina Marx <sabina.m...@sneo.io >> <mailto:sabina.m...@sneo.io>> wrote: >> > > >> > > > Yes, it's a new setup. The kafka and zookeeper runs as services >> and are >> > > > enabled, so they should start at system start. And kafka starts >> after >> > > > zookeeper. >> > > > And if I stop everything and start it, then it works. But then, >> if I >> > > > restart one server with zookeeper and kafka the kafka gets the >> > timeouts. >> > > > >> > > > Am 19.10.20, 18:16 schrieb "Mich Talebzadeh" < >> > mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com> >> > > >: >> > > > >> > > > OK so the issue seems to be kafka cluster. Is this a new >> setup? >> > > > >> > > > HTH >> > > > >> > > > >> > > > >> > > > LinkedIn * >> > > > >> > > >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > > > < >> > > > >> > > >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > > > >* >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > *Disclaimer:* Use it at your own risk. Any and all >> responsibility >> > for >> > > > any >> > > > loss, damage or destruction of data or any other property >> which may >> > > > arise >> > > > from relying on this email's technical content is explicitly >> > > > disclaimed. >> > > > The author will in no case be liable for any monetary >> damages >> > arising >> > > > from >> > > > such loss, damage or destruction. >> > > > >> > > > >> > > > >> > > > >> > > > On Mon, 19 Oct 2020 at 17:11, Sabina Marx < >> sabina.m...@sneo.io<mailto:sabina.m...@sneo.io>> >> > > wrote: >> > > > >> > > > > Thanks for your answer, but the zookeeper ensemble is >> running and >> > > > ports >> > > > > are ok. >> > > > > >> > > > > Am 19.10.20, 17:38 schrieb "Mich Talebzadeh" < >> > > > mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>>: >> > > > > >> > > > > Start the zookeeper ensemble first before starting >> Kafka >> > > > cluster. They >> > > > > need >> > > > > to select a leader and ensure that they all come >> online OK. >> > > > Check port >> > > > > 2181, 2888, 3888 using >> > > > > >> > > > > netstat -plten|egrep '2181|2888|3888' >> > > > > >> > > > > tcp 0 0 :::2181 :::* >> > > > > LISTEN 1005 9934134 29170/java >> > > > > tcp 0 0 ::ffff:50.140.197.217:2888< >> http://50.140.197.217:2888> :::* >> > > > > LISTEN 1005 9935496 29170/java >> > > > > tcp 0 0 ::ffff:50.140.197.217:3888< >> http://50.140.197.217:3888> :::* >> > > > > LISTEN 1005 9935493 29170/java >> > > > > >> > > > > >> > > > > >> > > > > P.S. I assume you are talking about Apache Kafks here. >> > > > > >> > > > > HTH >> > > > > >> > > > > >> > > > > >> > > > > LinkedIn * >> > > > > >> > > > >> > > >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > > > > < >> > > > > >> > > > >> > > >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > > > > >* >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > *Disclaimer:* Use it at your own risk. Any and all >> > > > responsibility for >> > > > > any >> > > > > loss, damage or destruction of data or any other >> property >> > which >> > > > may >> > > > > arise >> > > > > from relying on this email's technical content is >> explicitly >> > > > > disclaimed. >> > > > > The author will in no case be liable for any monetary >> damages >> > > > arising >> > > > > from >> > > > > such loss, damage or destruction. >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On Mon, 19 Oct 2020 at 14:42, Sabina Marx < >> > sabina.m...@sneo.io<mailto:sabina.m...@sneo.io> >> > > > >> > > > wrote: >> > > > > >> > > > > > Hi, >> > > > > > >> > > > > > I have a 5 nodes kafka cluster with 3 zookeepers. >> If I >> > > restart >> > > > 1 node >> > > > > > (zookeeper and kafka) the kafka gets a Client >> session timed >> > > > out, >> > > > > have not >> > > > > > heard from server in 6007ms for sessionid 0x0 >> > > > > > (org.apache.zookeeper.ClientCnxn) >> > > > > > Client session timed out, have not heard from >> server in >> > > 6007ms >> > > > for >> > > > > > sessionid 0x0, closing socket connection and >> attempting >> > > > reconnect >> > > > > > (org.apache.zookeeper.ClientCnxn) >> > > > > > And my kafka service do not start. >> > > > > > I have set the tickTime=6000 in the >> zookeeper.properties >> > but >> > > > that >> > > > > didn’t >> > > > > > help. What can I do? >> > > > > > >> > > > > > Many thanks for your help. >> > > > > > Sabina >> > > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > >> > >> >
