Hi, So, I started testing the steps suggested above to upgrade Kafka-Zk clusters with confluent-kafka-2.12 which will install v5.5.1 and was able to perform a rolling upgrade of Kafka clusters in a test env. And they are currently running with "inter.broker.protocol.version=1.1".
Before I switch it to 2.5, I wanted to perform a rolling upgrade of the ZK clusters and I am facing an error in bringing up the 1st follower node which I upgraded. I followed the below steps fyr: 1. Back up ZooKeeper data from the leader. Also, the zookeeper config files 2. Stopped the zookeeper service in follower node(one more is running along with a leader node) 3. Removed the previous version - sudo apt-get -y autoremove confluent-kafka-2.11 & sudo apt-get -y purge confluent-kafka-2.11 4. Add the 5.5 repo - sudo add-apt-repository "deb https://packages.confluent.io/deb/5.5 stable main" 5. Refreshed repo - sudo apt-get update 6. Installed the latest - sudo apt-get install confluent-kafka-2.12 7. Kept back the backed up configuration into /etc/kafka 8. Attempted to start zookeeper service which failed with below errors: [2020-09-01 14:04:18,909] INFO binding to port 0.0.0.0/0.0.0.0:xxxx (org.apache.zookeeper.server.NIOServerCnxnFactory) [2020-09-01 14:04:18,936] INFO Logging initialized @389ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [2020-09-01 14:04:19,101] WARN o.e.j.s.ServletContextHandler@15d0c81b{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler) [2020-09-01 14:04:19,101] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler) [2020-09-01 14:04:19,121] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-09-01 14:04:19,121] INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,121] INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,122] INFO tickTime set to 2000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,122] INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,122] INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,122] INFO initLimit set to 10 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,133] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [2020-09-01 14:04:19,134] INFO Using insecure (non-TLS) quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,134] INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,134] INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,134] INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-09-01 14:04:19,144] ERROR Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer) java.io.IOException: No snapshot found, but there are log entries. Something is broken! at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) [2020-09-01 14:04:19,146] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain) java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:938) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) Caused by: java.io.IOException: No snapshot found, but there are log entries. Something is broken! at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901) ... 4 more I compared the setup(configuration & log files)similar configurations are in leader node and observed that in leader node and the other follower node, there are no files in /var/log/kafka but in the updated ZK follower node which is failing to start has files written to /var/log/kafka Please help.. Thanks, Rijo Roy On 2020/08/19 17:52:34, Rijo Roy <royvk6...@gmail.com> wrote: > Sure Manoj! > > Really appreciate your quick response.. > > On 2020/08/19 17:40:54, <manoj.agraw...@cognizant.com> wrote: > > Great . > > Share your finding to this group once you done upgrade Confluent Kafka > > 4.1x to 5.3x successfully . > > > > I see many people having same question here . > > > > On 8/19/20, 10:38 AM, "Rijo Roy" <royvk6...@gmail.com> wrote: > > > > [External] > > > > > > Thanks Manoj! > > > > Yeah, the plan is to start with non-prod and validate first before > > going to prod. > > > > Thanks & Regards, > > Rijo Roy > > > > On 2020/08/19 17:33:53, <manoj.agraw...@cognizant.com> wrote: > > > I advise to do it non-prod for validation . > > > You can backup data log folder if you want but I have'nt see any > > issue . but better to backup data if it small . > > > > > > Don’t change below value to latest until you done full validation , > > once you changed to latest then you can't rollback . > > > > > > inter.broker.protocol.version=2.1.x > > > > > > On 8/19/20, 9:52 AM, "Rijo Roy" <royvk6...@gmail.com> wrote: > > > > > > [External] > > > > > > > > > Thanks Manoj! Appreciate your help.. > > > > > > I will follow the steps you pointed out.. > > > > > > Do you think there is a need to : > > > 1. backup the data before the rolling upgrade > > > 2. some kind of datasync that should be considered here.. I don't > > think this is required as I am performing an in-place upgrade.. > > > > > > Thanks & Regards, > > > Rijo Roy > > > > > > On 2020/08/18 20:45:42, <manoj.agraw...@cognizant.com> wrote: > > > > You can follow below steps > > > > > > > > 1. set inter.broker.protocol.version=2.1.x and rolling restart > > kafka > > > > 2. Rolling upgrade the Kafka cluster to 2.5 - > > > > 3. rolling upgrade ZK cluster > > > > Validate the kafka . > > > > > > > > 4. set inter.broker.protocol.version= new version and rolling > > restart the Kafka > > > > > > > > > > > > > > > > On 8/18/20, 12:54 PM, "Rijo Roy" <royvk6...@gmail.com> wrote: > > > > > > > > [External] > > > > > > > > > > > > Hi, > > > > > > > > I am a newbie in Kafka and would greatly appreciate if > > someone could help with best-practices and steps to upgrade to v5.3x. > > > > > > > > Below is my existing set-up: > > > > OS version: Ubuntu 16.04.6 LTS > > > > ZooKeeper version : 3.4.10 > > > > Kafka version : confluent-kafka-2.11 / 1.1.1-cp2 / v4.1.1 > > > > > > > > We need to upgrade our OS version to Ubuntu 18.04 LTS whose > > minimum requirement is to upgrade Kafka to v5.3x. Could someone please help > > me with the best-practices & steps for the upgrade.. > > > > > > > > Please let me know if you need any more information so that > > you could help me. > > > > > > > > Appreciate your help! > > > > > > > > Thanks & Regards, > > > > Rijo Roy > > > > > > > > > > > > > > > > This e-mail and any files transmitted with it are for the sole > > use of the intended recipient(s) and may contain confidential and > > privileged information. If you are not the intended recipient(s), please > > reply to the sender and destroy all copies of the original message. Any > > unauthorized review, use, disclosure, dissemination, forwarding, printing > > or copying of this email, and/or any action taken in reliance on the > > contents of this e-mail is strictly prohibited and may be unlawful. Where > > permitted by applicable law, this e-mail and other e-mail communications > > sent to and from Cognizant e-mail addresses may be monitored. > > > > This e-mail and any files transmitted with it are for the sole > > use of the intended recipient(s) and may contain confidential and > > privileged information. If you are not the intended recipient(s), please > > reply to the sender and destroy all copies of the original message. Any > > unauthorized review, use, disclosure, dissemination, forwarding, printing > > or copying of this email, and/or any action taken in reliance on the > > contents of this e-mail is strictly prohibited and may be unlawful. Where > > permitted by applicable law, this e-mail and other e-mail communications > > sent to and from Cognizant e-mail addresses may be monitored. > > > > > > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the intended recipient(s) and may contain confidential and privileged > > information. If you are not the intended recipient(s), please reply to the > > sender and destroy all copies of the original message. Any unauthorized > > review, use, disclosure, dissemination, forwarding, printing or copying of > > this email, and/or any action taken in reliance on the contents of this > > e-mail is strictly prohibited and may be unlawful. Where permitted by > > applicable law, this e-mail and other e-mail communications sent to and > > from Cognizant e-mail addresses may be monitored. > > > This e-mail and any files transmitted with it are for the sole use of > > the intended recipient(s) and may contain confidential and privileged > > information. If you are not the intended recipient(s), please reply to the > > sender and destroy all copies of the original message. Any unauthorized > > review, use, disclosure, dissemination, forwarding, printing or copying of > > this email, and/or any action taken in reliance on the contents of this > > e-mail is strictly prohibited and may be unlawful. Where permitted by > > applicable law, this e-mail and other e-mail communications sent to and > > from Cognizant e-mail addresses may be monitored. > > > > > > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. If you are not the intended recipient(s), please reply to the > > sender and destroy all copies of the original message. Any unauthorized > > review, use, disclosure, dissemination, forwarding, printing or copying of > > this email, and/or any action taken in reliance on the contents of this > > e-mail is strictly prohibited and may be unlawful. Where permitted by > > applicable law, this e-mail and other e-mail communications sent to and > > from Cognizant e-mail addresses may be monitored. > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. If you are not the intended recipient(s), please reply to the > > sender and destroy all copies of the original message. Any unauthorized > > review, use, disclosure, dissemination, forwarding, printing or copying of > > this email, and/or any action taken in reliance on the contents of this > > e-mail is strictly prohibited and may be unlawful. Where permitted by > > applicable law, this e-mail and other e-mail communications sent to and > > from Cognizant e-mail addresses may be monitored. > > >
