Hi Praveen, it seems to be the same problem, your log looks quite similar to mine. But I have no solution until now.
Regards Sabina Von: Praveen Kumar K S <prav...@securelyshare.com> Antworten an: "users@kafka.apache.org" <users@kafka.apache.org> Datum: Dienstag, 20. Oktober 2020 um 20:07 An: "users@kafka.apache.org" <users@kafka.apache.org> Betreff: Re: Client session timed out Hello, I'm not sure if I can add my issue in this thread. But it seems like I'm facing the same problem. KAFKA_VERSION=2.5.1 ZK_VERSION=3.5.8 I run 3 node zookeeper cluster and 3 node kafka cluster as docker containers in docker swarm environment. When I install it for first time, everything goes well. Zookeeper and Kafka are able to form the cluster. Services are healthy. But when I issue docker update command, kafka is not coming up though the zookeeper cluster is healthy. Below is the sequence of steps. docker service update one_zookeeper --image x.x.x/v1/zookeeper:latest --force docker service update one_zookeeper1 --image x.x.x/v1/zookeeper:latest --force docker service update one_zookeeper2 --image x.x.x/v1/zookeeper:latest --force Zookeeper is healthy now. I'm able to query leader and follower. Now, I'm updating kafka and it doesn't work. docker service update one_kafka --image x.x.com/v1/kafka:latest<http://x.x.com/v1/kafka:latest> --force PFA Kafka log. While kafka update has failed, I see that kafka1 and kafka2 are running and healthy. docker service ls | grep kafka one_kafka replicated 0/1 one_kafka1 replicated 1/1 one_kafka2 replicated 1/1 To cross verify, I have just brought down the services zookeeper and kafka without data loss. I preserve zookeeperdata,zookeeperlogs,zookeepertxns and kafkadata,kafkalogs. docker stack remove one docker stack deploy -c cluster-zookeeper.yml one docker stack deploy -c cluster-kafka.yml one Now, all the services are healthy. I'm not sure why kafka deployment is failing only during update. There is no change in the configuration in either zookeeper or kafka. Please help me resolve this issue and let me know if you need any additional details. Regards, Praveen Kumar K S +91-9986855625 On Tue, Oct 20, 2020 at 3:54 PM Sabina Marx <sabina.m...@sneo.io<mailto:sabina.m...@sneo.io>> wrote: Yes, it's the same problem. Am 19.10.20, 19:50 schrieb "Mich Talebzadeh" <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>>: can you try to disable automatic start and on the node just booted, start zookeeper first, check the log that it is connected and then start Kafka? I assume everything is set-up OK including in $KAFKA_HOME/config/server<N>.properties values for broker.id<http://broker.id>, hostname, zookeeper.connect=<server1>:2181,,server2>:2181, <serverN>:2181 and also zookeeper.connection.timeout.ms<http://zookeeper.connection.timeout.ms>=6000 (default) HTH LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 19 Oct 2020 at 18:17, Sabina Marx <sabina.m...@sneo.io<mailto:sabina.m...@sneo.io>> wrote: > Yes, you have it > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef> > ________________________________ > Von: Mich Talebzadeh <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>> > Gesendet: Monday, October 19, 2020 7:09:53 PM > An: users@kafka.apache.org<mailto:users@kafka.apache.org> <users@kafka.apache.org<mailto:users@kafka.apache.org>> > Betreff: Re: Client session timed out > > Ok I think it is clearer now. > > As I understand all your Zookeepers and Kafkas are running. (5 nodes > meaning 5 physical hosts?). Then you have to reboot one physical host. You > still have the redundancy. But when the physical host comes up and your > zookeeper and then Kafka come up, you have Kafka timing out and not > connecting to the existing Kafka cluster? > > Does that make sense? > > > > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > < > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >* > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 19 Oct 2020 at 17:59, Sabina Marx <sabina.m...@sneo.io<mailto:sabina.m...@sneo.io>> wrote: > > > No, sorry I‘m not so good in explaining. > > The scenario is: the complete Cluster is running, all zookeepers and all > > kafkas. And then I restart one server, the others are still running. > > > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef> > > ________________________________ > > Von: Mich Talebzadeh <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>> > > Gesendet: Monday, October 19, 2020 6:46:49 PM > > An: users@kafka.apache.org<mailto:users@kafka.apache.org> <users@kafka.apache.org<mailto:users@kafka.apache.org>> > > Betreff: Re: Client session timed out > > > > Can you pls clarify when you say you start one pair (Zookeeper and > Kafka), > > what happens to the others. Do you keep them down? > > > > > > > > > > LinkedIn * > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > < > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > >* > > > > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > > loss, damage or destruction of data or any other property which may arise > > from relying on this email's technical content is explicitly disclaimed. > > The author will in no case be liable for any monetary damages arising > from > > such loss, damage or destruction. > > > > > > > > > > On Mon, 19 Oct 2020 at 17:33, Sabina Marx <sabina.m...@sneo.io<mailto:sabina.m...@sneo.io>> wrote: > > > > > Yes, it's a new setup. The kafka and zookeeper runs as services and are > > > enabled, so they should start at system start. And kafka starts after > > > zookeeper. > > > And if I stop everything and start it, then it works. But then, if I > > > restart one server with zookeeper and kafka the kafka gets the > timeouts. > > > > > > Am 19.10.20, 18:16 schrieb "Mich Talebzadeh" < > mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com> > > >: > > > > > > OK so the issue seems to be kafka cluster. Is this a new setup? > > > > > > HTH > > > > > > > > > > > > LinkedIn * > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > < > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > >* > > > > > > > > > > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility > for > > > any > > > loss, damage or destruction of data or any other property which may > > > arise > > > from relying on this email's technical content is explicitly > > > disclaimed. > > > The author will in no case be liable for any monetary damages > arising > > > from > > > such loss, damage or destruction. > > > > > > > > > > > > > > > On Mon, 19 Oct 2020 at 17:11, Sabina Marx <sabina.m...@sneo.io<mailto:sabina.m...@sneo.io>> > > wrote: > > > > > > > Thanks for your answer, but the zookeeper ensemble is running and > > > ports > > > > are ok. > > > > > > > > Am 19.10.20, 17:38 schrieb "Mich Talebzadeh" < > > > mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>>: > > > > > > > > Start the zookeeper ensemble first before starting Kafka > > > cluster. They > > > > need > > > > to select a leader and ensure that they all come online OK. > > > Check port > > > > 2181, 2888, 3888 using > > > > > > > > netstat -plten|egrep '2181|2888|3888' > > > > > > > > tcp 0 0 :::2181 :::* > > > > LISTEN 1005 9934134 29170/java > > > > tcp 0 0 ::ffff:50.140.197.217:2888<http://50.140.197.217:2888> :::* > > > > LISTEN 1005 9935496 29170/java > > > > tcp 0 0 ::ffff:50.140.197.217:3888<http://50.140.197.217:3888> :::* > > > > LISTEN 1005 9935493 29170/java > > > > > > > > > > > > > > > > P.S. I assume you are talking about Apache Kafks here. > > > > > > > > HTH > > > > > > > > > > > > > > > > LinkedIn * > > > > > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > < > > > > > > > > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > >* > > > > > > > > > > > > > > > > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all > > > responsibility for > > > > any > > > > loss, damage or destruction of data or any other property > which > > > may > > > > arise > > > > from relying on this email's technical content is explicitly > > > > disclaimed. > > > > The author will in no case be liable for any monetary damages > > > arising > > > > from > > > > such loss, damage or destruction. > > > > > > > > > > > > > > > > > > > > On Mon, 19 Oct 2020 at 14:42, Sabina Marx < > sabina.m...@sneo.io<mailto:sabina.m...@sneo.io> > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I have a 5 nodes kafka cluster with 3 zookeepers. If I > > restart > > > 1 node > > > > > (zookeeper and kafka) the kafka gets a Client session timed > > > out, > > > > have not > > > > > heard from server in 6007ms for sessionid 0x0 > > > > > (org.apache.zookeeper.ClientCnxn) > > > > > Client session timed out, have not heard from server in > > 6007ms > > > for > > > > > sessionid 0x0, closing socket connection and attempting > > > reconnect > > > > > (org.apache.zookeeper.ClientCnxn) > > > > > And my kafka service do not start. > > > > > I have set the tickTime=6000 in the zookeeper.properties > but > > > that > > > > didn’t > > > > > help. What can I do? > > > > > > > > > > Many thanks for your help. > > > > > Sabina > > > > > > > > > > > > > > > > > > > > > >
