Hi Team, I am running ansible to install elk on one server. (on Dev env, standalone setup) but my playbook is getting failed at kafka role for task listing of Kafka topics. Can you please help. below are the logs.
1. playbook getting failed with error: - ``` TASK [kafka : Kafka Topics | List topics] ****************************************************************************************************************************** fatal: [kafka_1]: FAILED! => { "changed": false, "cmd": [ "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh", "--zookeeper", "<FQDN>:5181", "--list" ], "delta": "0:00:34.084478", "end": "2020-11-19 22:33:42.624353", "rc": 1, "start": "2020-11-19 22:33:08.539875" } STDERR: Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) ``` 2. Getting same error when run command manually which playbook is trying to run. ``` /opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper <ip of localhost>:5181 --list Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262) at kafka.admin.TopicCommand$.main(TopicCommand.scala:53) at kafka.admin.TopicCommand.main(TopicCommand.scala) ``` 3. both Kafka and zookeeper are failed and not getting started. ``` systemctl status kafka.service -l ● kafka.service Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 IST; 5min ago Process: 5448 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh (code=exited, status=1/FAILURE) Process: 4742 ExecStart=/bin/sh -c /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/server.properties > /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 4742 (code=exited, status=1/FAILURE) Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service. Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1 Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state. Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed. systemctl status zookeeper.service -l ● zookeeper.service Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 IST; 5min ago Process: 5281 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh (code=exited, status=1/FAILURE) Process: 4448 ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (code=exited, status=1/FAILURE) Main PID: 4448 (code=exited, status=1/FAILURE) Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) Nov 19 22:33:09 systemd[1]: zookeeper.service: main process exited, code=exited, status=1/FAILURE Nov 19 22:33:09 systemd[1]: zookeeper.service: control process exited, code=exited status=1 Nov 19 22:33:09 systemd[1]: Unit zookeeper.service entered failed state. Nov 19 22:33:09 systemd[1]: zookeeper.service failed. ``` 4. therefore unable to do telnet or netstat telnet <ip og localhost> 5181 Trying 10.. telnet: connect to address <ip of localhost>: Connection refused netstat -tnlp |grep 5181 No output No Iptables or SELinux is running. 5. output of /opt/kafka/kafka_2.12-2.2.2/kafka.log [2020-11-19 22:33:09,853] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [2020-11-19 22:33:10,584] INFO starting (kafka.server.KafkaServer) [2020-11-19 22:33:10,585] INFO Connecting to zookeeper on <ip of localhost>:5181 (kafka.server.KafkaServer) [2020-11-19 22:33:10,640] INFO [ZooKeeperClient] Initializing a new session to <ip of localhost>:5181. (kafka.zookeeper.ZooKeeperClient) [2020-11-19 22:33:10,646] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,646] INFO Client environment:host.name=<FQDN> (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,646] INFO Client environment:java.version=1.8.0_221 (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,646] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,646] INFO Client environment:java.home=/opt/jre1.8.0_221 (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,646] INFO Client environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-api-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-basic-auth-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-file-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-json-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-runtime-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/ka fka_2.12-2.2.2/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-core-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-databind-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/.. /libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-http-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-io-9.4.14.v20181114.jar:/opt/kafk a/kafka_2.12-2.2.2/bin/../libs/jetty-security-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-server-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlet-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlets-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-examples-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafka_2. 12-2.2.2/bin/../libs/kafka-tools-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/log4j-1.2.17.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/lz4-java-1.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/maven-artifact-3.6.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/reflections-0.9.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/rocksdbjni-5.15.10.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0 .Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,646] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,646] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,647] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,647] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,647] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,647] INFO Client environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,647] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,647] INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,647] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,648] INFO Initiating client connection, connectString=<ip of localhost>:5181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3c419631 (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:10,667] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:10,673] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:10,676] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [2020-11-19 22:33:11,796] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:11,797] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:12,897] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:12,898] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:13,999] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:13,999] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:15,101] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:15,101] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:16,202] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:16,202] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:16,678] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient) [2020-11-19 22:33:17,303] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:17,406] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [2020-11-19 22:33:17,407] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn) [2020-11-19 22:33:17,409] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient) [2020-11-19 22:33:17,412] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238) at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825) at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361) at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385) at kafka.server.KafkaServer.startup(KafkaServer.scala:205) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) at kafka.Kafka$.main(Kafka.scala:75) at kafka.Kafka.main(Kafka.scala) [2020-11-19 22:33:17,415] INFO shutting down (kafka.server.KafkaServer) [2020-11-19 22:33:17,422] INFO shut down completed (kafka.server.KafkaServer) [2020-11-19 22:33:17,422] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable) [2020-11-19 22:33:17,426] INFO shutting down (kafka.server.KafkaServer) 6. /var/log/messages Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_ 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_ 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_ 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.941+0530#011INFO#011kafka/log.go:53#011client/metadata fetching metadata for [heartbeat] from bro ker <ip of localhost>:3997 <ip of localhost>:3997: connect: connection refused 19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/metadata got error from broker -1 while fetching me tadata: dial tcp <ip of localhost>:3997: connect: connection refused Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011kafka message: client/metadata no available broker to send metadata request to Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011[publisher]#011pipeline/retry.go:155#011Drop batch Nov 19 18:16:07 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:07,962] INFO Reading configuration from: /opt/kafka/kafka_2.12-2.2.2/config/zookeep er.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost> (org.apache.zookeeper.server.quorum.QuorumPeer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009] ERROR Invalid configuration, only one server specified (ignoring) (org.apac he.zookeeper.server.quorum.QuorumPeerConfig) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.Datadi rCleanupManager) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirC leanupManager) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Reading configuration from: /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost> (org.apache.zookeeper.server.quorum.QuorumPeer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] ERROR Invalid configuration, only one server specified (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:host.name=<FQDN> (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.version=1.8.0_221 (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.home=/opt/jre1.8.0_221 (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12 -2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar: 2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.name=kafka (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.home=/home/kafka (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.dir=/ (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,044] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,049] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,054] INFO binding to port 0.0.0.0/0.0.0.0:5181 (org.apache.zookeeper.server.NIOServerCnxnFactory) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,131] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: java.io.EOFException Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at java.io.DataInputStream.readInt(DataInputStream.java:392) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: main process exited, code=exited, status=1/FAILURE Nov 19 18:16:08 <HOSTNAME> zookeeper-server-stop.sh: No zookeeper server to stop Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: control process exited, code=exited status=1 Nov 19 18:16:08 <HOSTNAME> systemd: Unit zookeeper.service entered failed state. Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service failed. 7. config file - **/opt/kafka/kafka_2.12-2.2.2/config/server.properties** broker.id=1 advertised.listeners=PLAINTEXT://hostname of localhost:3997 listeners=PLAINTEXT://hostname of localhost:3997 delete.topic.enable=true auto.create.topics.enable=True offsets.topic.replication.factor=1 # The number of threads handling network requests num.network.threads=3 # The number of threads doing disk I/O num.io.threads=6 # The send buffer (SO_SNDBUF) used by the socket server #socket.send.buffer.bytes=102400 # The receive buffer (SO_RCVBUF) used by the socket server #socket.receive.buffer.bytes=102400 # The maximum size of a request that the socket server will accept (protection against OOM) #socket.request.max.bytes=104857600 # A comma seperated list of directories under which to store log files log.dirs=/var/log/kafka # The default number of log partitions per topic. num.partitions=3 # Default Replication Factor default.replication.factor=1 # The minimum age of a log file to be eligible for deletion log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log as long as the remaining # segments don't drop below log.retention.bytes. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=1073741824 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms=300000 zookeeper.connect= <ip of localhost>:5181 zookeeper.connection.timeout.ms=6000 **/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties** # clientPortBindAddress=localhost clientPort=5181 dataDir=/var/data/zookeeper dataLogDir=/var/data/logs/zookeeper maxClientCnxns=0 # time for inital synchronization initLimit=10 # how many ticks can pass before timeout syncLimit=5 server.1=<hostname of localhost>:2888:3888 8. journalctl -fu zookeeper -- Logs begin at Thu 2020-11-19 01:05:17 IST. -- Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122) Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89) Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55) Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119) Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: main process exited, code=exited, status=1/FAILURE Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: control process exited, code=exited status=1 Nov 19 22:33:09 <HOSTNAME> systemd[1]: Unit zookeeper.service entered failed state. Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service failed. journalctl -fu kafka.service -- Logs begin at Thu 2020-11-19 01:05:17 IST. -- Nov 19 22:23:30 <HOSTNAME> systemd[1]: Started kafka.service. Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1 Nov 19 22:23:40 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state. Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service failed. Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service. Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1 Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state. Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed. this is the single server where everything is installed of elk along with tomcat etc.. so I have stopped other services like tomcat for better resource management. hope resource crunch is not the issue. free -g total used free shared buff/cache available Mem: 15 7 6 0 1 7 Swap: 3 0 3 Thanks,