Hello Experts, Any help to debug and resolve this issue is highly appreciated.
Regards, Praveen On Wed, 21 Oct, 2020, 11:26 Sabina Marx, <sabina.m...@sneo.io> wrote: > Hi Praveen, > > it seems to be the same problem, your log looks quite similar to mine. But > I have no solution until now. > > Regards > Sabina > > Von: Praveen Kumar K S <prav...@securelyshare.com> > Antworten an: "users@kafka.apache.org" <users@kafka.apache.org> > Datum: Dienstag, 20. Oktober 2020 um 20:07 > An: "users@kafka.apache.org" <users@kafka.apache.org> > Betreff: Re: Client session timed out > > Hello, > > I'm not sure if I can add my issue in this thread. But it seems like I'm > facing the same problem. > > KAFKA_VERSION=2.5.1 > ZK_VERSION=3.5.8 > > I run 3 node zookeeper cluster and 3 node kafka cluster as docker > containers in docker swarm environment. When I install it for first time, > everything goes well. Zookeeper and Kafka are able to form the cluster. > Services are healthy. > > But when I issue docker update command, kafka is not coming up though the > zookeeper cluster is healthy. Below is the sequence of steps. > > docker service update one_zookeeper --image x.x.x/v1/zookeeper:latest > --force > docker service update one_zookeeper1 --image x.x.x/v1/zookeeper:latest > --force > docker service update one_zookeeper2 --image x.x.x/v1/zookeeper:latest > --force > > Zookeeper is healthy now. I'm able to query leader and follower. > > Now, I'm updating kafka and it doesn't work. > docker service update one_kafka --image x.x.com/v1/kafka:latest< > http://x.x.com/v1/kafka:latest> --force > > PFA Kafka log. > > While kafka update has failed, I see that kafka1 and kafka2 are running > and healthy. > > docker service ls | grep kafka > one_kafka replicated 0/1 > one_kafka1 replicated 1/1 > one_kafka2 replicated 1/1 > > To cross verify, I have just brought down the services zookeeper and kafka > without data loss. I preserve zookeeperdata,zookeeperlogs,zookeepertxns and > kafkadata,kafkalogs. > > docker stack remove one > docker stack deploy -c cluster-zookeeper.yml one > docker stack deploy -c cluster-kafka.yml one > > Now, all the services are healthy. > > I'm not sure why kafka deployment is failing only during update. There is > no change in the configuration in either zookeeper or kafka. > > Please help me resolve this issue and let me know if you need any > additional details. > > Regards, > Praveen Kumar K S > +91-9986855625 > > > On Tue, Oct 20, 2020 at 3:54 PM Sabina Marx <sabina.m...@sneo.io<mailto: > sabina.m...@sneo.io>> wrote: > Yes, it's the same problem. > > Am 19.10.20, 19:50 schrieb "Mich Talebzadeh" <mich.talebza...@gmail.com > <mailto:mich.talebza...@gmail.com>>: > > can you try to disable automatic start and on the node just booted, > start > zookeeper first, check the log that it is connected and then start > Kafka? > > I assume everything is set-up OK including in > $KAFKA_HOME/config/server<N>.properties values for broker.id< > http://broker.id>, hostname, > zookeeper.connect=<server1>:2181,,server2>:2181, <serverN>:2181 and > also > > zookeeper.connection.timeout.ms<http://zookeeper.connection.timeout.ms>=6000 > (default) > > HTH > > > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > < > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >* > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for > any > loss, damage or destruction of data or any other property which may > arise > from relying on this email's technical content is explicitly > disclaimed. > The author will in no case be liable for any monetary damages arising > from > such loss, damage or destruction. > > > > > On Mon, 19 Oct 2020 at 18:17, Sabina Marx <sabina.m...@sneo.io<mailto: > sabina.m...@sneo.io>> wrote: > > > Yes, you have it > > > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef> > > ________________________________ > > Von: Mich Talebzadeh <mich.talebza...@gmail.com<mailto: > mich.talebza...@gmail.com>> > > Gesendet: Monday, October 19, 2020 7:09:53 PM > > An: users@kafka.apache.org<mailto:users@kafka.apache.org> < > users@kafka.apache.org<mailto:users@kafka.apache.org>> > > Betreff: Re: Client session timed out > > > > Ok I think it is clearer now. > > > > As I understand all your Zookeepers and Kafkas are running. (5 nodes > > meaning 5 physical hosts?). Then you have to reboot one physical > host. You > > still have the redundancy. But when the physical host comes up and > your > > zookeeper and then Kafka come up, you have Kafka timing out and not > > connecting to the existing Kafka cluster? > > > > Does that make sense? > > > > > > > > > > > > > > LinkedIn * > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > < > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > >* > > > > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility > for any > > loss, damage or destruction of data or any other property which may > arise > > from relying on this email's technical content is explicitly > disclaimed. > > The author will in no case be liable for any monetary damages > arising from > > such loss, damage or destruction. > > > > > > > > > > On Mon, 19 Oct 2020 at 17:59, Sabina Marx <sabina.m...@sneo.io > <mailto:sabina.m...@sneo.io>> wrote: > > > > > No, sorry I‘m not so good in explaining. > > > The scenario is: the complete Cluster is running, all zookeepers > and all > > > kafkas. And then I restart one server, the others are still > running. > > > > > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef> > > > ________________________________ > > > Von: Mich Talebzadeh <mich.talebza...@gmail.com<mailto: > mich.talebza...@gmail.com>> > > > Gesendet: Monday, October 19, 2020 6:46:49 PM > > > An: users@kafka.apache.org<mailto:users@kafka.apache.org> < > users@kafka.apache.org<mailto:users@kafka.apache.org>> > > > Betreff: Re: Client session timed out > > > > > > Can you pls clarify when you say you start one pair (Zookeeper and > > Kafka), > > > what happens to the others. Do you keep them down? > > > > > > > > > > > > > > > LinkedIn * > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > < > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > >* > > > > > > > > > > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility > for any > > > loss, damage or destruction of data or any other property which > may arise > > > from relying on this email's technical content is explicitly > disclaimed. > > > The author will in no case be liable for any monetary damages > arising > > from > > > such loss, damage or destruction. > > > > > > > > > > > > > > > On Mon, 19 Oct 2020 at 17:33, Sabina Marx <sabina.m...@sneo.io > <mailto:sabina.m...@sneo.io>> wrote: > > > > > > > Yes, it's a new setup. The kafka and zookeeper runs as services > and are > > > > enabled, so they should start at system start. And kafka starts > after > > > > zookeeper. > > > > And if I stop everything and start it, then it works. But then, > if I > > > > restart one server with zookeeper and kafka the kafka gets the > > timeouts. > > > > > > > > Am 19.10.20, 18:16 schrieb "Mich Talebzadeh" < > > mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com> > > > >: > > > > > > > > OK so the issue seems to be kafka cluster. Is this a new > setup? > > > > > > > > HTH > > > > > > > > > > > > > > > > LinkedIn * > > > > > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > < > > > > > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > >* > > > > > > > > > > > > > > > > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all > responsibility > > for > > > > any > > > > loss, damage or destruction of data or any other property > which may > > > > arise > > > > from relying on this email's technical content is explicitly > > > > disclaimed. > > > > The author will in no case be liable for any monetary damages > > arising > > > > from > > > > such loss, damage or destruction. > > > > > > > > > > > > > > > > > > > > On Mon, 19 Oct 2020 at 17:11, Sabina Marx < > sabina.m...@sneo.io<mailto:sabina.m...@sneo.io>> > > > wrote: > > > > > > > > > Thanks for your answer, but the zookeeper ensemble is > running and > > > > ports > > > > > are ok. > > > > > > > > > > Am 19.10.20, 17:38 schrieb "Mich Talebzadeh" < > > > > mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>>: > > > > > > > > > > Start the zookeeper ensemble first before starting > Kafka > > > > cluster. They > > > > > need > > > > > to select a leader and ensure that they all come > online OK. > > > > Check port > > > > > 2181, 2888, 3888 using > > > > > > > > > > netstat -plten|egrep '2181|2888|3888' > > > > > > > > > > tcp 0 0 :::2181 :::* > > > > > LISTEN 1005 9934134 29170/java > > > > > tcp 0 0 ::ffff:50.140.197.217:2888< > http://50.140.197.217:2888> :::* > > > > > LISTEN 1005 9935496 29170/java > > > > > tcp 0 0 ::ffff:50.140.197.217:3888< > http://50.140.197.217:3888> :::* > > > > > LISTEN 1005 9935493 29170/java > > > > > > > > > > > > > > > > > > > > P.S. I assume you are talking about Apache Kafks here. > > > > > > > > > > HTH > > > > > > > > > > > > > > > > > > > > LinkedIn * > > > > > > > > > > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > > < > > > > > > > > > > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > > >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all > > > > responsibility for > > > > > any > > > > > loss, damage or destruction of data or any other > property > > which > > > > may > > > > > arise > > > > > from relying on this email's technical content is > explicitly > > > > > disclaimed. > > > > > The author will in no case be liable for any monetary > damages > > > > arising > > > > > from > > > > > such loss, damage or destruction. > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 19 Oct 2020 at 14:42, Sabina Marx < > > sabina.m...@sneo.io<mailto:sabina.m...@sneo.io> > > > > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I have a 5 nodes kafka cluster with 3 zookeepers. If > I > > > restart > > > > 1 node > > > > > > (zookeeper and kafka) the kafka gets a Client > session timed > > > > out, > > > > > have not > > > > > > heard from server in 6007ms for sessionid 0x0 > > > > > > (org.apache.zookeeper.ClientCnxn) > > > > > > Client session timed out, have not heard from server > in > > > 6007ms > > > > for > > > > > > sessionid 0x0, closing socket connection and > attempting > > > > reconnect > > > > > > (org.apache.zookeeper.ClientCnxn) > > > > > > And my kafka service do not start. > > > > > > I have set the tickTime=6000 in the > zookeeper.properties > > but > > > > that > > > > > didn’t > > > > > > help. What can I do? > > > > > > > > > > > > Many thanks for your help. > > > > > > Sabina > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
