Hi Sunil, Thanks for your reply.
all four words commands where giving connection refused. I moved files (log.1,log.103,log.142,log.16a,log.17d - all where having oct timestamp) present inside `/var/data/logs/zookeeper/version-2/` to some other directory and started zookeeper and then kafka and both started and are up and running. I am able to list the topics now. now I am see log.1 file got created. (its binary so can't see inside it) Can any one know the reason behind it? why it worked after moving these log* files which seems to be little old. not sure if same has to do after 1 month from now if same issue occurs again. @Sasikumar, Thanks. Thanks, On 2020/11/22 12:43:12, sunil chaudhari <sunilmchaudhar...@gmail.com> wrote: > Hi, > You can refer to the documentation for logging levels. > > > https://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#Debug+Log+Cleanup+%28log4j%29 > > By clearing current logs will help you to read and understand new logs. > > Old logs wont get mixed up with the new one and you can read and debug > events from the beginning of startup. > > Regards, > Sunil. > > On Sun, 22 Nov 2020 at 5:52 PM, prat 007 <pratkul...@gmail.com> wrote: > > > Hi Sunil, > > > > Thanks for your reply. > > You mean to say to empty kafka.log file? (I don't see zookeeper.log file > > in server), but not sure how clearing the log file itself will help. > > > > I will search for debug switch for zookeeper start command but I was > > talking about in which zookeeper config file I can change the value of a > > parameter from INFO to DEBUG or TRACE level to enable the whole log file to > > show the logs in more depth. > > > > thanks, > > > > On 2020/11/22 12:00:26, sunil chaudhari <sunilmchaudhar...@gmail.com> > > wrote: > > > Hi I am talking about the Error : > > > > > > 18:16:08,027] ERROR Invalid configuration, only one server specified > > > > > (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > > > > > > > > > This seems to be the problem of whole cluster. > > > > > > Clear all logs means, clear existing server logs. Not data. > > > > > > You can see help of the zookeeper start command for debug switch. > > > > > > Regards, > > > Sunil. > > > > > > > > > On Sun, 22 Nov 2020 at 5:00 PM, prat 007 <pratkul...@gmail.com> wrote: > > > > > > > Hi Sunil, > > > > > > > > Thanks for reply. Q. Are you pointing to below line from point 6 to be > > > > similar with url you have given (in Troubleshooting, under section: > > Server > > > > not coming up because of file corruption)? > > > > > > > > ov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: > > java.io.EOFException > > > > > > > > Q. `Clear all logs` - you mean to say delete all files from > > > > `datadir/version-2 and datalogdir/version-2/`? > > > > If yes, is it safe to delete them. Do we require them later ? > > > > > > > > the url says, > > > > `make sure all the other servers in your ensemble are up and working. > > Use > > > > "stat" command on the command port to see if they are in good health`. > > > > > > > > Q. Is below command correct to verify good health ? > > > > # echo stat | nc localhost 2181 > > > > > > > > Q. Can you please let me know where can I set zookeeper logs for > > verbose > > > > mode for debugging the issue? > > > > > > > > Thanks, > > > > > > > > On 2020/11/22 01:26:52, sunil chaudhari <sunilmchaudhar...@gmail.com> > > > > wrote: > > > > > Hi, > > > > > Please check if it helps: > > > > > > > > > > > http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_supportedPlatforms > > > > > > > > > > You have similar symptoms in point 6 in your original email. > > > > > > > > > > Try running it manually with below steps: > > > > > Clear all logs > > > > > Start zookeeper manually, watch logs for any error. Rectify it > > referring > > > > > above link. > > > > > Start kafka manually. Watch info/error logs. > > > > > Then try to list topics. > > > > > > > > > > Regards, > > > > > Sunil. > > > > > > > > > > > > > > > On Sat, 21 Nov 2020 at 10:16 PM, prat 007 <pratkul...@gmail.com> > > wrote: > > > > > > > > > > > Hi Team, > > > > > > > > > > > > I am running ansible to install elk on one server. (on Dev env, > > > > standalone > > > > > > setup) but my playbook is getting failed at kafka role for task > > > > listing of > > > > > > Kafka topics. Can you please help. > > > > > > below are the logs. > > > > > > > > > > > > 1. > > > > > > playbook getting failed with error: - > > > > > > ``` > > > > > > TASK [kafka : Kafka Topics | List topics] > > > > > > > > > > > > ****************************************************************************************************************************** > > > > > > fatal: [kafka_1]: FAILED! => { > > > > > > "changed": false, > > > > > > "cmd": [ > > > > > > "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh", > > > > > > "--zookeeper", > > > > > > "<FQDN>:5181", > > > > > > "--list" > > > > > > ], > > > > > > "delta": "0:00:34.084478", > > > > > > "end": "2020-11-19 22:33:42.624353", > > > > > > "rc": 1, > > > > > > "start": "2020-11-19 22:33:08.539875" > > > > > > } > > > > > > > > > > > > STDERR: > > > > > > > > > > > > Exception in thread "main" > > > > > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting > > for > > > > > > connection while in state: CONNECTING > > > > > > at > > > > > > > > > > > > kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) > > > > > > at > > > > > > > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > > > > > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > > > > > > at > > > > > > > > > > > > kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) > > > > > > at > > > > kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) > > > > > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) > > > > > > ``` > > > > > > > > > > > > 2. Getting same error when run command manually which playbook is > > > > trying > > > > > > to run. > > > > > > > > > > > > ``` > > > > > > /opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper <ip of > > > > > > localhost>:5181 --list > > > > > > > > > > > > Exception in thread "main" > > > > > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting > > for > > > > > > connection while in state: CONNECTING > > > > > > at > > > > > > > > > > > > kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) > > > > > > at > > > > > > > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > > > > > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > > > > > > at > > > > > > > > > > > > kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) > > > > > > at > > > > kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) > > > > > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) > > > > > > at > > > > > > > > > > > > kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262) > > > > > > at kafka.admin.TopicCommand$.main(TopicCommand.scala:53) > > > > > > at kafka.admin.TopicCommand.main(TopicCommand.scala) > > > > > > ``` > > > > > > > > > > > > 3. both Kafka and zookeeper are failed and not getting started. > > > > > > > > > > > > ``` > > > > > > systemctl status kafka.service -l > > > > > > ● kafka.service > > > > > > Loaded: loaded (/etc/systemd/system/kafka.service; enabled; > > vendor > > > > > > preset: disabled) > > > > > > Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 > > > > IST; > > > > > > 5min ago > > > > > > Process: 5448 > > > > > > ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh > > > > (code=exited, > > > > > > status=1/FAILURE) > > > > > > Process: 4742 ExecStart=/bin/sh -c > > > > > > /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh > > > > > > /opt/kafka/kafka_2.12-2.2.2/config/server.properties > > > > > > > /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, > > > > status=1/FAILURE) > > > > > > Main PID: 4742 (code=exited, status=1/FAILURE) > > > > > > > > > > > > Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service. > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process > > > > exited, > > > > > > code=exited, status=1/FAILURE > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control > > process > > > > > > exited, code=exited status=1 > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered > > > > failed > > > > > > state. > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed. > > > > > > > > > > > > > > > > > > systemctl status zookeeper.service -l > > > > > > ● zookeeper.service > > > > > > Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; > > > > vendor > > > > > > preset: disabled) > > > > > > Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 > > > > IST; > > > > > > 5min ago > > > > > > Process: 5281 > > > > > > ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh > > > > > > (code=exited, status=1/FAILURE) > > > > > > Process: 4448 > > > > > > ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh > > > > > > /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties > > (code=exited, > > > > > > status=1/FAILURE) > > > > > > Main PID: 4448 (code=exited, status=1/FAILURE) > > > > > > > > > > > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) > > > > > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) > > > > > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) > > > > > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) > > > > > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) > > > > > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) > > > > > > Nov 19 22:33:09 systemd[1]: zookeeper.service: main process exited, > > > > > > code=exited, status=1/FAILURE > > > > > > Nov 19 22:33:09 systemd[1]: zookeeper.service: control process > > exited, > > > > > > code=exited status=1 > > > > > > Nov 19 22:33:09 systemd[1]: Unit zookeeper.service entered failed > > > > state. > > > > > > Nov 19 22:33:09 systemd[1]: zookeeper.service failed. > > > > > > ``` > > > > > > > > > > > > 4. therefore unable to do telnet or netstat > > > > > > > > > > > > telnet <ip og localhost> 5181 > > > > > > Trying 10.. > > > > > > telnet: connect to address <ip of localhost>: Connection refused > > > > > > > > > > > > netstat -tnlp |grep 5181 > > > > > > No output > > > > > > > > > > > > No Iptables or SELinux is running. > > > > > > > > > > > > 5. output of /opt/kafka/kafka_2.12-2.2.2/kafka.log > > > > > > > > > > > > [2020-11-19 22:33:09,853] INFO Registered > > > > kafka:type=kafka.Log4jController > > > > > > MBean (kafka.utils.Log4jControllerRegistration$) > > > > > > [2020-11-19 22:33:10,584] INFO starting (kafka.server.KafkaServer) > > > > > > [2020-11-19 22:33:10,585] INFO Connecting to zookeeper on <ip of > > > > > > localhost>:5181 (kafka.server.KafkaServer) > > > > > > [2020-11-19 22:33:10,640] INFO [ZooKeeperClient] Initializing a new > > > > > > session to <ip of localhost>:5181. > > (kafka.zookeeper.ZooKeeperClient) > > > > > > [2020-11-19 22:33:10,646] INFO Client > > > > > > > > > > > > environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, > > > > > > built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,646] INFO Client environment:host.name=<FQDN> > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,646] INFO Client > > > > environment:java.version=1.8.0_221 > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,646] INFO Client > > environment:java.vendor=Oracle > > > > > > Corporation (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,646] INFO Client > > > > > > environment:java.home=/opt/jre1.8.0_221 > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,646] INFO Client > > > > > > > > > > > > environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-api-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-basic-auth-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-file-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-json-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-runtime-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/ka > > > > > > > > > > > > > > > > > > fka_2.12-2.2.2/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-core-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-databind-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bi > > > > n/.. > > > > > > > > > > > > > > > > > > /libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/ > > > > > > javax.ws > > > > > > > > > > > > .rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-http-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-io-9.4.14.v20181114.jar:/opt/kafk > > > > > > > > > > > > > > > > > > a/kafka_2.12-2.2.2/bin/../libs/jetty-security-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-server-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlet-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlets-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-examples-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafk > > > > a_2. > > > > > > > > > > > > > > > > > > 12-2.2.2/bin/../libs/kafka-tools-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/log4j-1.2.17.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/lz4-java-1.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/maven-artifact-3.6.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/reflections-0.9.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/rocksdbjni-5.15.10.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1 > > > > .1.0 > > > > > > > > > > > > .Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,646] INFO Client > > > > > > > > > > > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,646] INFO Client > > environment:java.io.tmpdir=/tmp > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,647] INFO Client > > environment:java.compiler=<NA> > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,647] INFO Client environment:os.name=Linux > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,647] INFO Client environment:os.arch=amd64 > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,647] INFO Client > > > > > > environment:os.version=3.10.0-1062.el7.x86_64 > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,647] INFO Client environment:user.name=kafka > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,647] INFO Client > > environment:user.home=/home/kafka > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,647] INFO Client environment:user.dir=/ > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,648] INFO Initiating client connection, > > > > > > connectString=<ip of localhost>:5181 sessionTimeout=6000 > > > > > > > > > > > > watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3c419631 > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:10,667] INFO Opening socket connection to server > > > > > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate > > using > > > > SASL > > > > > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:10,673] INFO Socket error occurred: <FQDN>/<ip of > > > > > > localhost>:5181: Connection refused > > (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:10,676] INFO [ZooKeeperClient] Waiting until > > > > connected. > > > > > > (kafka.zookeeper.ZooKeeperClient) > > > > > > [2020-11-19 22:33:11,796] INFO Opening socket connection to server > > > > > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate > > using > > > > SASL > > > > > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:11,797] INFO Socket error occurred: <FQDN>/<ip of > > > > > > localhost>:5181: Connection refused > > (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:12,897] INFO Opening socket connection to server > > > > > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate > > using > > > > SASL > > > > > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:12,898] INFO Socket error occurred: <FQDN>/<ip of > > > > > > localhost>:5181: Connection refused > > (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:13,999] INFO Opening socket connection to server > > > > > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate > > using > > > > SASL > > > > > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:13,999] INFO Socket error occurred: <FQDN>/<ip of > > > > > > localhost>:5181: Connection refused > > (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:15,101] INFO Opening socket connection to server > > > > > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate > > using > > > > SASL > > > > > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:15,101] INFO Socket error occurred: <FQDN>/<ip of > > > > > > localhost>:5181: Connection refused > > (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:16,202] INFO Opening socket connection to server > > > > > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate > > using > > > > SASL > > > > > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:16,202] INFO Socket error occurred: <FQDN>/<ip of > > > > > > localhost>:5181: Connection refused > > (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:16,678] INFO [ZooKeeperClient] Closing. > > > > > > (kafka.zookeeper.ZooKeeperClient) > > > > > > [2020-11-19 22:33:17,303] INFO Opening socket connection to server > > > > > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate > > using > > > > SASL > > > > > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:17,406] INFO Session: 0x0 closed > > > > > > (org.apache.zookeeper.ZooKeeper) > > > > > > [2020-11-19 22:33:17,407] INFO EventThread shut down for session: > > 0x0 > > > > > > (org.apache.zookeeper.ClientCnxn) > > > > > > [2020-11-19 22:33:17,409] INFO [ZooKeeperClient] Closed. > > > > > > (kafka.zookeeper.ZooKeeperClient) > > > > > > [2020-11-19 22:33:17,412] ERROR Fatal error during KafkaServer > > startup. > > > > > > Prepare to shutdown (kafka.server.KafkaServer) > > > > > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting > > for > > > > > > connection while in state: CONNECTING > > > > > > at > > > > > > > > > > > > kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) > > > > > > at > > > > > > > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > > > > > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > > > > > > at > > > > > > > > > > > > kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) > > > > > > at > > > > kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) > > > > > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) > > > > > > at > > > > kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361) > > > > > > at > > kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385) > > > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:205) > > > > > > at > > > > > > > > > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) > > > > > > at kafka.Kafka$.main(Kafka.scala:75) > > > > > > at kafka.Kafka.main(Kafka.scala) > > > > > > [2020-11-19 22:33:17,415] INFO shutting down > > (kafka.server.KafkaServer) > > > > > > [2020-11-19 22:33:17,422] INFO shut down completed > > > > > > (kafka.server.KafkaServer) > > > > > > [2020-11-19 22:33:17,422] ERROR Exiting Kafka. > > > > > > (kafka.server.KafkaServerStartable) > > > > > > [2020-11-19 22:33:17,426] INFO shutting down > > (kafka.server.KafkaServer) > > > > > > > > > > > > 6. /var/log/messages > > > > > > > > > > > > Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with > > creates=None > > > > > > executable=None _uses_shell=False strip_empty_ends=True > > > > > > _raw_params=/opt/kafka/kafka_ > > > > > > 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list > > > > removes=None > > > > > > argv=None warn=True chdir=None stdin_add_newline=True stdin=None > > > > > > Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with > > creates=None > > > > > > executable=None _uses_shell=False strip_empty_ends=True > > > > > > _raw_params=/opt/kafka/kafka_ > > > > > > 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list > > > > removes=None > > > > > > argv=None warn=True chdir=None stdin_add_newline=True stdin=None > > > > > > Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with > > creates=None > > > > > > executable=None _uses_shell=False strip_empty_ends=True > > > > > > _raw_params=/opt/kafka/kafka_ > > > > > > 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list > > > > removes=None > > > > > > argv=None warn=True chdir=None stdin_add_newline=True stdin=None > > > > > > Nov 19 18:16:07 <HOSTNAME> heartbeat: > > > > > > > > > > > > 2020-11-19T18:16:07.941+0530#011INFO#011kafka/log.go:53#011client/metadata > > > > > > fetching metadata for [heartbeat] from bro > > > > > > ker <ip of localhost>:3997 > > > > > > <ip of localhost>:3997: connect: connection refused > > > > > > 19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/metadata > > got > > > > > > error from broker -1 while fetching me > > > > > > tadata: dial tcp <ip of localhost>:3997: connect: connection > > refused > > > > > > Nov 19 18:16:07 <HOSTNAME> heartbeat: > > > > > > 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011kafka > > > > message: > > > > > > client/metadata no available broker to send > > > > > > metadata request to > > > > > > Nov 19 18:16:07 <HOSTNAME> heartbeat: > > > > > > > > > > > > 2020-11-19T18:16:07.942+0530#011INFO#011[publisher]#011pipeline/retry.go:155#011Drop > > > > > > batch > > > > > > Nov 19 18:16:07 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:07,962] INFO Reading configuration from: > > > > > > /opt/kafka/kafka_2.12-2.2.2/config/zookeep > > > > > > er.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,009] INFO Resolved hostname: <FQDN> to address: > > <FQDN>/<ip of > > > > > > localhost> (org.apache.zookeeper.server.quorum.QuorumPeer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,009] ERROR Invalid configuration, only one server > > specified > > > > > > (ignoring) (org.apac > > > > > > he.zookeeper.server.quorum.QuorumPeerConfig) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,011] INFO autopurge.snapRetainCount set to 3 > > > > > > (org.apache.zookeeper.server.Datadi > > > > > > rCleanupManager) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,011] INFO autopurge.purgeInterval set to 0 > > > > > > (org.apache.zookeeper.server.DatadirC > > > > > > leanupManager) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,011] INFO Purge task is not scheduled. > > > > > > (org.apache.zookeeper.server.DatadirCleanupManager) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,011] WARN Either no config or no quorum defined in config, > > > > > > running in standalone mode > > > > > > (org.apache.zookeeper.server.quorum.QuorumPeerMain > > > > > > > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,027] INFO Reading configuration from: > > > > > > /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties > > > > > > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,027] INFO Resolved hostname: <FQDN> to address: > > <FQDN>/<ip of > > > > > > localhost> (org.apache.zookeeper.server.quorum.QuorumPeer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,027] ERROR Invalid configuration, only one server > > specified > > > > > > (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,027] INFO Starting server > > > > > > (org.apache.zookeeper.server.ZooKeeperServerMain) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,033] INFO Server > > > > > > > > > > > > environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, > > > > > > built on 06/29/2018 00:39 GMT > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,033] INFO Server environment:host.name=<FQDN> > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,033] INFO Server environment:java.version=1.8.0_221 > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,033] INFO Server environment:java.vendor=Oracle > > Corporation > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,033] INFO Server environment:java.home=/opt/jre1.8.0_221 > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,033] INFO Server > > > > > > > > > > > > environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12 > > > > > > > > > > > > > > > > > > -2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar: > > > > > > > > > > > > 2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server > > > > > > > > > > > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server environment:java.io.tmpdir=/tmp > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server environment:java.compiler=<NA> > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server environment:os.name=Linux > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server environment:os.arch=amd64 > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server > > environment:os.version=3.10.0-1062.el7.x86_64 > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server environment:user.name=kafka > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server environment:user.home=/home/kafka > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,034] INFO Server environment:user.dir=/ > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,043] INFO tickTime set to 3000 > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,043] INFO minSessionTimeout set to -1 > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,044] INFO maxSessionTimeout set to -1 > > > > > > (org.apache.zookeeper.server.ZooKeeperServer) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,049] INFO Using > > > > org.apache.zookeeper.server.NIOServerCnxnFactory > > > > > > as server connection factory > > > > (org.apache.zookeeper.server.ServerCnxnFactory) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,054] INFO binding to port 0.0.0.0/0.0.0.0:5181 > > > > > > (org.apache.zookeeper.server.NIOServerCnxnFactory) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > > > > > 18:16:08,131] ERROR Unexpected exception, exiting abnormally > > > > > > (org.apache.zookeeper.server.ZooKeeperServerMain) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: > > > > java.io.EOFException > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > java.io.DataInputStream.readInt(DataInputStream.java:392) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) > > > > > > Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: main process > > > > > > exited, code=exited, status=1/FAILURE > > > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-stop.sh: No zookeeper > > > > server > > > > > > to stop > > > > > > Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: control > > process > > > > > > exited, code=exited status=1 > > > > > > Nov 19 18:16:08 <HOSTNAME> systemd: Unit zookeeper.service entered > > > > failed > > > > > > state. > > > > > > Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service failed. > > > > > > > > > > > > 7. config file - > > > > > > > > > > > > **/opt/kafka/kafka_2.12-2.2.2/config/server.properties** > > > > > > > > > > > > broker.id=1 > > > > > > advertised.listeners=PLAINTEXT://hostname of localhost:3997 > > > > > > listeners=PLAINTEXT://hostname of localhost:3997 > > > > > > > > > > > > delete.topic.enable=true > > > > > > auto.create.topics.enable=True > > > > > > offsets.topic.replication.factor=1 > > > > > > > > > > > > # The number of threads handling network requests > > > > > > num.network.threads=3 > > > > > > > > > > > > # The number of threads doing disk I/O > > > > > > num.io.threads=6 > > > > > > > > > > > > # The send buffer (SO_SNDBUF) used by the socket server > > > > > > #socket.send.buffer.bytes=102400 > > > > > > > > > > > > # The receive buffer (SO_RCVBUF) used by the socket server > > > > > > #socket.receive.buffer.bytes=102400 > > > > > > > > > > > > # The maximum size of a request that the socket server will accept > > > > > > (protection against OOM) > > > > > > #socket.request.max.bytes=104857600 > > > > > > > > > > > > # A comma seperated list of directories under which to store log > > files > > > > > > log.dirs=/var/log/kafka > > > > > > > > > > > > # The default number of log partitions per topic. > > > > > > num.partitions=3 > > > > > > > > > > > > # Default Replication Factor > > > > > > default.replication.factor=1 > > > > > > > > > > > > # The minimum age of a log file to be eligible for deletion > > > > > > log.retention.hours=168 > > > > > > > > > > > > # A size-based retention policy for logs. Segments are pruned from > > the > > > > log > > > > > > as long as the remaining > > > > > > # segments don't drop below log.retention.bytes. > > > > > > #log.retention.bytes=1073741824 > > > > > > > > > > > > # The maximum size of a log segment file. When this size is > > reached a > > > > new > > > > > > log segment will be created. > > > > > > log.segment.bytes=1073741824 > > > > > > > > > > > > # The interval at which log segments are checked to see if they > > can be > > > > > > deleted according > > > > > > # to the retention policies > > > > > > log.retention.check.interval.ms=300000 > > > > > > > > > > > > zookeeper.connect= <ip of localhost>:5181 > > > > > > zookeeper.connection.timeout.ms=6000 > > > > > > > > > > > > > > > > > > **/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties** > > > > > > > > > > > > # clientPortBindAddress=localhost > > > > > > clientPort=5181 > > > > > > > > > > > > dataDir=/var/data/zookeeper > > > > > > dataLogDir=/var/data/logs/zookeeper > > > > > > maxClientCnxns=0 > > > > > > > > > > > > # time for inital synchronization > > > > > > initLimit=10 > > > > > > # how many ticks can pass before timeout > > > > > > syncLimit=5 > > > > > > > > > > > > server.1=<hostname of localhost>:2888:3888 > > > > > > > > > > > > 8. > > > > > > > > > > > > journalctl -fu zookeeper > > > > > > -- Logs begin at Thu 2020-11-19 01:05:17 IST. -- > > > > > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) > > > > > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) > > > > > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) > > > > > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) > > > > > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) > > > > > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) > > > > > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: main > > process > > > > > > exited, code=exited, status=1/FAILURE > > > > > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: control > > > > process > > > > > > exited, code=exited status=1 > > > > > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: Unit zookeeper.service > > entered > > > > > > failed state. > > > > > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service failed. > > > > > > > > > > > > journalctl -fu kafka.service > > > > > > -- Logs begin at Thu 2020-11-19 01:05:17 IST. -- > > > > > > Nov 19 22:23:30 <HOSTNAME> systemd[1]: Started kafka.service. > > > > > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: main process > > > > exited, > > > > > > code=exited, status=1/FAILURE > > > > > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: control > > process > > > > > > exited, code=exited status=1 > > > > > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: Unit kafka.service entered > > > > failed > > > > > > state. > > > > > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service failed. > > > > > > Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service. > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process > > > > exited, > > > > > > code=exited, status=1/FAILURE > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control > > process > > > > > > exited, code=exited status=1 > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered > > > > failed > > > > > > state. > > > > > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed. > > > > > > > > > > > > this is the single server where everything is installed of elk > > along > > > > with > > > > > > tomcat etc.. so I have stopped other services like tomcat for > > better > > > > > > resource management. hope resource crunch is not the issue. > > > > > > > > > > > > free -g > > > > > > total used free shared buff/cache available > > > > > > Mem: 15 7 6 0 1 7 > > > > > > Swap: 3 0 3 > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > >
