Hi Sunil, Thanks for reply. Q. Are you pointing to below line from point 6 to be similar with url you have given (in Troubleshooting, under section: Server not coming up because of file corruption)?
ov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: java.io.EOFException Q. `Clear all logs` - you mean to say delete all files from `datadir/version-2 and datalogdir/version-2/`? If yes, is it safe to delete them. Do we require them later ? the url says, `make sure all the other servers in your ensemble are up and working. Use "stat" command on the command port to see if they are in good health`. Q. Is below command correct to verify good health ? # echo stat | nc localhost 2181 Q. Can you please let me know where can I set zookeeper logs for verbose mode for debugging the issue? Thanks, On 2020/11/22 01:26:52, sunil chaudhari <sunilmchaudhar...@gmail.com> wrote: > Hi, > Please check if it helps: > http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_supportedPlatforms > > You have similar symptoms in point 6 in your original email. > > Try running it manually with below steps: > Clear all logs > Start zookeeper manually, watch logs for any error. Rectify it referring > above link. > Start kafka manually. Watch info/error logs. > Then try to list topics. > > Regards, > Sunil. > > > On Sat, 21 Nov 2020 at 10:16 PM, prat 007 <pratkul...@gmail.com> wrote: > > > Hi Team, > > > > I am running ansible to install elk on one server. (on Dev env, standalone > > setup) but my playbook is getting failed at kafka role for task listing of > > Kafka topics. Can you please help. > > below are the logs. > > > > 1. > > playbook getting failed with error: - > > ``` > > TASK [kafka : Kafka Topics | List topics] > > ****************************************************************************************************************************** > > fatal: [kafka_1]: FAILED! => { > > "changed": false, > > "cmd": [ > > "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh", > > "--zookeeper", > > "<FQDN>:5181", > > "--list" > > ], > > "delta": "0:00:34.084478", > > "end": "2020-11-19 22:33:42.624353", > > "rc": 1, > > "start": "2020-11-19 22:33:08.539875" > > } > > > > STDERR: > > > > Exception in thread "main" > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for > > connection while in state: CONNECTING > > at > > kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) > > at > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > > at > > kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) > > at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) > > ``` > > > > 2. Getting same error when run command manually which playbook is trying > > to run. > > > > ``` > > /opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper <ip of > > localhost>:5181 --list > > > > Exception in thread "main" > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for > > connection while in state: CONNECTING > > at > > kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) > > at > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > > at > > kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) > > at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) > > at > > kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262) > > at kafka.admin.TopicCommand$.main(TopicCommand.scala:53) > > at kafka.admin.TopicCommand.main(TopicCommand.scala) > > ``` > > > > 3. both Kafka and zookeeper are failed and not getting started. > > > > ``` > > systemctl status kafka.service -l > > ● kafka.service > > Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor > > preset: disabled) > > Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 IST; > > 5min ago > > Process: 5448 > > ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh (code=exited, > > status=1/FAILURE) > > Process: 4742 ExecStart=/bin/sh -c > > /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh > > /opt/kafka/kafka_2.12-2.2.2/config/server.properties > > > /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, status=1/FAILURE) > > Main PID: 4742 (code=exited, status=1/FAILURE) > > > > Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service. > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, > > code=exited, status=1/FAILURE > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process > > exited, code=exited status=1 > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed > > state. > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed. > > > > > > systemctl status zookeeper.service -l > > ● zookeeper.service > > Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor > > preset: disabled) > > Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 IST; > > 5min ago > > Process: 5281 > > ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh > > (code=exited, status=1/FAILURE) > > Process: 4448 > > ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh > > /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (code=exited, > > status=1/FAILURE) > > Main PID: 4448 (code=exited, status=1/FAILURE) > > > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) > > Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) > > Nov 19 22:33:09 systemd[1]: zookeeper.service: main process exited, > > code=exited, status=1/FAILURE > > Nov 19 22:33:09 systemd[1]: zookeeper.service: control process exited, > > code=exited status=1 > > Nov 19 22:33:09 systemd[1]: Unit zookeeper.service entered failed state. > > Nov 19 22:33:09 systemd[1]: zookeeper.service failed. > > ``` > > > > 4. therefore unable to do telnet or netstat > > > > telnet <ip og localhost> 5181 > > Trying 10.. > > telnet: connect to address <ip of localhost>: Connection refused > > > > netstat -tnlp |grep 5181 > > No output > > > > No Iptables or SELinux is running. > > > > 5. output of /opt/kafka/kafka_2.12-2.2.2/kafka.log > > > > [2020-11-19 22:33:09,853] INFO Registered kafka:type=kafka.Log4jController > > MBean (kafka.utils.Log4jControllerRegistration$) > > [2020-11-19 22:33:10,584] INFO starting (kafka.server.KafkaServer) > > [2020-11-19 22:33:10,585] INFO Connecting to zookeeper on <ip of > > localhost>:5181 (kafka.server.KafkaServer) > > [2020-11-19 22:33:10,640] INFO [ZooKeeperClient] Initializing a new > > session to <ip of localhost>:5181. (kafka.zookeeper.ZooKeeperClient) > > [2020-11-19 22:33:10,646] INFO Client > > environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, > > built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,646] INFO Client environment:host.name=<FQDN> > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,646] INFO Client environment:java.version=1.8.0_221 > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,646] INFO Client environment:java.vendor=Oracle > > Corporation (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,646] INFO Client > > environment:java.home=/opt/jre1.8.0_221 (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,646] INFO Client > > environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-api-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-basic-auth-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-file-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-json-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-runtime-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/ka > > > > > > fka_2.12-2.2.2/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-core-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-databind-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bi n/.. > > > > > > /libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/ > > javax.ws > > .rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-http-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-io-9.4.14.v20181114.jar:/opt/kafk > > > > > > a/kafka_2.12-2.2.2/bin/../libs/jetty-security-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-server-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlet-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlets-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-examples-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafk a_2. > > > > > > 12-2.2.2/bin/../libs/kafka-tools-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/log4j-1.2.17.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/lz4-java-1.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/maven-artifact-3.6.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/reflections-0.9.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/rocksdbjni-5.15.10.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1 .1.0 > > > > .Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,646] INFO Client > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,646] INFO Client environment:java.io.tmpdir=/tmp > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,647] INFO Client environment:java.compiler=<NA> > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,647] INFO Client environment:os.name=Linux > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,647] INFO Client environment:os.arch=amd64 > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,647] INFO Client > > environment:os.version=3.10.0-1062.el7.x86_64 > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,647] INFO Client environment:user.name=kafka > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,647] INFO Client environment:user.home=/home/kafka > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,647] INFO Client environment:user.dir=/ > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,648] INFO Initiating client connection, > > connectString=<ip of localhost>:5181 sessionTimeout=6000 > > watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3c419631 > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:10,667] INFO Opening socket connection to server > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:10,673] INFO Socket error occurred: <FQDN>/<ip of > > localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:10,676] INFO [ZooKeeperClient] Waiting until connected. > > (kafka.zookeeper.ZooKeeperClient) > > [2020-11-19 22:33:11,796] INFO Opening socket connection to server > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:11,797] INFO Socket error occurred: <FQDN>/<ip of > > localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:12,897] INFO Opening socket connection to server > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:12,898] INFO Socket error occurred: <FQDN>/<ip of > > localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:13,999] INFO Opening socket connection to server > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:13,999] INFO Socket error occurred: <FQDN>/<ip of > > localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:15,101] INFO Opening socket connection to server > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:15,101] INFO Socket error occurred: <FQDN>/<ip of > > localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:16,202] INFO Opening socket connection to server > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:16,202] INFO Socket error occurred: <FQDN>/<ip of > > localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:16,678] INFO [ZooKeeperClient] Closing. > > (kafka.zookeeper.ZooKeeperClient) > > [2020-11-19 22:33:17,303] INFO Opening socket connection to server > > <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL > > (unknown error) (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:17,406] INFO Session: 0x0 closed > > (org.apache.zookeeper.ZooKeeper) > > [2020-11-19 22:33:17,407] INFO EventThread shut down for session: 0x0 > > (org.apache.zookeeper.ClientCnxn) > > [2020-11-19 22:33:17,409] INFO [ZooKeeperClient] Closed. > > (kafka.zookeeper.ZooKeeperClient) > > [2020-11-19 22:33:17,412] ERROR Fatal error during KafkaServer startup. > > Prepare to shutdown (kafka.server.KafkaServer) > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for > > connection while in state: CONNECTING > > at > > kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) > > at > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > > at > > kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) > > at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) > > at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361) > > at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385) > > at kafka.server.KafkaServer.startup(KafkaServer.scala:205) > > at > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) > > at kafka.Kafka$.main(Kafka.scala:75) > > at kafka.Kafka.main(Kafka.scala) > > [2020-11-19 22:33:17,415] INFO shutting down (kafka.server.KafkaServer) > > [2020-11-19 22:33:17,422] INFO shut down completed > > (kafka.server.KafkaServer) > > [2020-11-19 22:33:17,422] ERROR Exiting Kafka. > > (kafka.server.KafkaServerStartable) > > [2020-11-19 22:33:17,426] INFO shutting down (kafka.server.KafkaServer) > > > > 6. /var/log/messages > > > > Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None > > executable=None _uses_shell=False strip_empty_ends=True > > _raw_params=/opt/kafka/kafka_ > > 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None > > argv=None warn=True chdir=None stdin_add_newline=True stdin=None > > Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None > > executable=None _uses_shell=False strip_empty_ends=True > > _raw_params=/opt/kafka/kafka_ > > 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None > > argv=None warn=True chdir=None stdin_add_newline=True stdin=None > > Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None > > executable=None _uses_shell=False strip_empty_ends=True > > _raw_params=/opt/kafka/kafka_ > > 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None > > argv=None warn=True chdir=None stdin_add_newline=True stdin=None > > Nov 19 18:16:07 <HOSTNAME> heartbeat: > > 2020-11-19T18:16:07.941+0530#011INFO#011kafka/log.go:53#011client/metadata > > fetching metadata for [heartbeat] from bro > > ker <ip of localhost>:3997 > > <ip of localhost>:3997: connect: connection refused > > 19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/metadata got > > error from broker -1 while fetching me > > tadata: dial tcp <ip of localhost>:3997: connect: connection refused > > Nov 19 18:16:07 <HOSTNAME> heartbeat: > > 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011kafka message: > > client/metadata no available broker to send > > metadata request to > > Nov 19 18:16:07 <HOSTNAME> heartbeat: > > 2020-11-19T18:16:07.942+0530#011INFO#011[publisher]#011pipeline/retry.go:155#011Drop > > batch > > Nov 19 18:16:07 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:07,962] INFO Reading configuration from: > > /opt/kafka/kafka_2.12-2.2.2/config/zookeep > > er.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,009] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of > > localhost> (org.apache.zookeeper.server.quorum.QuorumPeer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,009] ERROR Invalid configuration, only one server specified > > (ignoring) (org.apac > > he.zookeeper.server.quorum.QuorumPeerConfig) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,011] INFO autopurge.snapRetainCount set to 3 > > (org.apache.zookeeper.server.Datadi > > rCleanupManager) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,011] INFO autopurge.purgeInterval set to 0 > > (org.apache.zookeeper.server.DatadirC > > leanupManager) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,011] INFO Purge task is not scheduled. > > (org.apache.zookeeper.server.DatadirCleanupManager) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,011] WARN Either no config or no quorum defined in config, > > running in standalone mode > > (org.apache.zookeeper.server.quorum.QuorumPeerMain > > > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,027] INFO Reading configuration from: > > /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties > > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,027] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of > > localhost> (org.apache.zookeeper.server.quorum.QuorumPeer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,027] ERROR Invalid configuration, only one server specified > > (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,027] INFO Starting server > > (org.apache.zookeeper.server.ZooKeeperServerMain) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,033] INFO Server > > environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, > > built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,033] INFO Server environment:host.name=<FQDN> > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,033] INFO Server environment:java.version=1.8.0_221 > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,033] INFO Server environment:java.vendor=Oracle Corporation > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,033] INFO Server environment:java.home=/opt/jre1.8.0_221 > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,033] INFO Server > > environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12 > > > > > > -2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar: > > 2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:java.io.tmpdir=/tmp > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:java.compiler=<NA> > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:os.name=Linux > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:os.arch=amd64 > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:os.version=3.10.0-1062.el7.x86_64 > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:user.name=kafka > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:user.home=/home/kafka > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,034] INFO Server environment:user.dir=/ > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,043] INFO tickTime set to 3000 > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,043] INFO minSessionTimeout set to -1 > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,044] INFO maxSessionTimeout set to -1 > > (org.apache.zookeeper.server.ZooKeeperServer) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,049] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory > > as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,054] INFO binding to port 0.0.0.0/0.0.0.0:5181 > > (org.apache.zookeeper.server.NIOServerCnxnFactory) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 > > 18:16:08,131] ERROR Unexpected exception, exiting abnormally > > (org.apache.zookeeper.server.ZooKeeperServerMain) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: java.io.EOFException > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > java.io.DataInputStream.readInt(DataInputStream.java:392) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) > > Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: main process > > exited, code=exited, status=1/FAILURE > > Nov 19 18:16:08 <HOSTNAME> zookeeper-server-stop.sh: No zookeeper server > > to stop > > Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: control process > > exited, code=exited status=1 > > Nov 19 18:16:08 <HOSTNAME> systemd: Unit zookeeper.service entered failed > > state. > > Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service failed. > > > > 7. config file - > > > > **/opt/kafka/kafka_2.12-2.2.2/config/server.properties** > > > > broker.id=1 > > advertised.listeners=PLAINTEXT://hostname of localhost:3997 > > listeners=PLAINTEXT://hostname of localhost:3997 > > > > delete.topic.enable=true > > auto.create.topics.enable=True > > offsets.topic.replication.factor=1 > > > > # The number of threads handling network requests > > num.network.threads=3 > > > > # The number of threads doing disk I/O > > num.io.threads=6 > > > > # The send buffer (SO_SNDBUF) used by the socket server > > #socket.send.buffer.bytes=102400 > > > > # The receive buffer (SO_RCVBUF) used by the socket server > > #socket.receive.buffer.bytes=102400 > > > > # The maximum size of a request that the socket server will accept > > (protection against OOM) > > #socket.request.max.bytes=104857600 > > > > # A comma seperated list of directories under which to store log files > > log.dirs=/var/log/kafka > > > > # The default number of log partitions per topic. > > num.partitions=3 > > > > # Default Replication Factor > > default.replication.factor=1 > > > > # The minimum age of a log file to be eligible for deletion > > log.retention.hours=168 > > > > # A size-based retention policy for logs. Segments are pruned from the log > > as long as the remaining > > # segments don't drop below log.retention.bytes. > > #log.retention.bytes=1073741824 > > > > # The maximum size of a log segment file. When this size is reached a new > > log segment will be created. > > log.segment.bytes=1073741824 > > > > # The interval at which log segments are checked to see if they can be > > deleted according > > # to the retention policies > > log.retention.check.interval.ms=300000 > > > > zookeeper.connect= <ip of localhost>:5181 > > zookeeper.connection.timeout.ms=6000 > > > > > > **/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties** > > > > # clientPortBindAddress=localhost > > clientPort=5181 > > > > dataDir=/var/data/zookeeper > > dataLogDir=/var/data/logs/zookeeper > > maxClientCnxns=0 > > > > # time for inital synchronization > > initLimit=10 > > # how many ticks can pass before timeout > > syncLimit=5 > > > > server.1=<hostname of localhost>:2888:3888 > > > > 8. > > > > journalctl -fu zookeeper > > -- Logs begin at Thu 2020-11-19 01:05:17 IST. -- > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) > > Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: main process > > exited, code=exited, status=1/FAILURE > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: control process > > exited, code=exited status=1 > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: Unit zookeeper.service entered > > failed state. > > Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service failed. > > > > journalctl -fu kafka.service > > -- Logs begin at Thu 2020-11-19 01:05:17 IST. -- > > Nov 19 22:23:30 <HOSTNAME> systemd[1]: Started kafka.service. > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: main process exited, > > code=exited, status=1/FAILURE > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: control process > > exited, code=exited status=1 > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: Unit kafka.service entered failed > > state. > > Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service failed. > > Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service. > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, > > code=exited, status=1/FAILURE > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process > > exited, code=exited status=1 > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed > > state. > > Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed. > > > > this is the single server where everything is installed of elk along with > > tomcat etc.. so I have stopped other services like tomcat for better > > resource management. hope resource crunch is not the issue. > > > > free -g > > total used free shared buff/cache available > > Mem: 15 7 6 0 1 7 > > Swap: 3 0 3 > > > > Thanks, > > >
