Hai,
Looks the host is not reachable.
Can you try tcptraceroute or telnet?
Thanks SasiKumar Senthilnath
On 2020/11/21 12:39:40 prat 007 wrote:
> Hi Team,
>
> I am running ansible to install elk on one server. (on Dev env, standalone
> setup) but my playbook is getting failed at kafka role for task listing of
> Kafka topics. Can you please help.
> below are the logs.
>
> 1.
> playbook getting failed with error: -
> ```
> TASK [kafka : Kafka Topics | List topics]
> ******************************************************************************************************************************
> fatal: [kafka_1]: FAILED! => {
> "changed": false,
> "cmd": [
> "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh",
> "--zookeeper",
> "<FQDN>:5181",
> "--list"
> ],
> "delta": "0:00:34.084478",
> "end": "2020-11-19 22:33:42.624353",
> "rc": 1,
> "start": "2020-11-19 22:33:08.539875"
> }
>
> STDERR:
>
> Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException:
> Timed out waiting for connection while in state: CONNECTING
> at
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> at
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> ```
>
> 2. Getting same error when run command manually which playbook is trying to
> run.
>
> ```
> /opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper <ip of
> localhost>:5181 --list
>
> Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException:
> Timed out waiting for connection while in state: CONNECTING
> at
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> at
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> at
> kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262)
> at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
> at kafka.admin.TopicCommand.main(TopicCommand.scala)
> ```
>
> 3. both Kafka and zookeeper are failed and not getting started.
>
> ```
> systemctl status kafka.service -l
> ● kafka.service
> Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset:
> disabled)
> Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 IST; 5min
> ago
> Process: 5448 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh
> (code=exited, status=1/FAILURE)
> Process: 4742 ExecStart=/bin/sh -c
> /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh
> /opt/kafka/kafka_2.12-2.2.2/config/server.properties >
> /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, status=1/FAILURE)
> Main PID: 4742 (code=exited, status=1/FAILURE)
>
> Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited,
> code=exited, status=1/FAILURE
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited,
> code=exited status=1
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed
> state.
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.
>
>
> systemctl status zookeeper.service -l
> ● zookeeper.service
> Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor
> preset: disabled)
> Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 IST; 5min
> ago
> Process: 5281
> ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh
> (code=exited, status=1/FAILURE)
> Process: 4448
> ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh
> /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (code=exited,
> status=1/FAILURE)
> Main PID: 4448 (code=exited, status=1/FAILURE)
>
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
> Nov 19 22:33:09 systemd[1]: zookeeper.service: main process exited,
> code=exited, status=1/FAILURE
> Nov 19 22:33:09 systemd[1]: zookeeper.service: control process exited,
> code=exited status=1
> Nov 19 22:33:09 systemd[1]: Unit zookeeper.service entered failed state.
> Nov 19 22:33:09 systemd[1]: zookeeper.service failed.
> ```
>
> 4. therefore unable to do telnet or netstat
>
> telnet <ip og localhost> 5181
> Trying 10..
> telnet: connect to address <ip of localhost>: Connection refused
>
> netstat -tnlp |grep 5181
> No output
>
> No Iptables or SELinux is running.
>
> 5. output of /opt/kafka/kafka_2.12-2.2.2/kafka.log
>
> [2020-11-19 22:33:09,853] INFO Registered kafka:type=kafka.Log4jController
> MBean (kafka.utils.Log4jControllerRegistration$)
> [2020-11-19 22:33:10,584] INFO starting (kafka.server.KafkaServer)
> [2020-11-19 22:33:10,585] INFO Connecting to zookeeper on <ip of
> localhost>:5181 (kafka.server.KafkaServer)
> [2020-11-19 22:33:10,640] INFO [ZooKeeperClient] Initializing a new session
> to <ip of localhost>:5181. (kafka.zookeeper.ZooKeeperClient)
> [2020-11-19 22:33:10,646] INFO Client
> environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03,
> built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,646] INFO Client environment:host.name=<FQDN>
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,646] INFO Client environment:java.version=1.8.0_221
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,646] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,646] INFO Client environment:java.home=/opt/jre1.8.0_221
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,646] INFO Client
> environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-api-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-basic-auth-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-file-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-json-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-runtime-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/
ka
>
> fka_2.12-2.2.2/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-core-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-databind-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/
..
>
> /libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-http-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-io-9.4.14.v20181114.jar:/opt/ka
fk
>
> a/kafka_2.12-2.2.2/bin/../libs/jetty-security-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-server-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlet-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlets-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-examples-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafka_
2.
>
> 12-2.2.2/bin/../libs/kafka-tools-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/log4j-1.2.17.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/lz4-java-1.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/maven-artifact-3.6.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/reflections-0.9.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/rocksdbjni-5.15.10.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1
.0
>
> .Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,646] INFO Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,646] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,647] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,647] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,647] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,647] INFO Client
> environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,647] INFO Client environment:user.name=kafka
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,647] INFO Client environment:user.home=/home/kafka
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,647] INFO Client environment:user.dir=/
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,648] INFO Initiating client connection,
> connectString=<ip of localhost>:5181 sessionTimeout=6000
> watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3c419631
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:10,667] INFO Opening socket connection to server <FQDN>/<ip
> of localhost>:5181. Will not attempt to authenticate using SASL (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:10,673] INFO Socket error occurred: <FQDN>/<ip of
> localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:10,676] INFO [ZooKeeperClient] Waiting until connected.
> (kafka.zookeeper.ZooKeeperClient)
> [2020-11-19 22:33:11,796] INFO Opening socket connection to server <FQDN>/<ip
> of localhost>:5181. Will not attempt to authenticate using SASL (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:11,797] INFO Socket error occurred: <FQDN>/<ip of
> localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:12,897] INFO Opening socket connection to server <FQDN>/<ip
> of localhost>:5181. Will not attempt to authenticate using SASL (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:12,898] INFO Socket error occurred: <FQDN>/<ip of
> localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:13,999] INFO Opening socket connection to server <FQDN>/<ip
> of localhost>:5181. Will not attempt to authenticate using SASL (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:13,999] INFO Socket error occurred: <FQDN>/<ip of
> localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:15,101] INFO Opening socket connection to server <FQDN>/<ip
> of localhost>:5181. Will not attempt to authenticate using SASL (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:15,101] INFO Socket error occurred: <FQDN>/<ip of
> localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:16,202] INFO Opening socket connection to server <FQDN>/<ip
> of localhost>:5181. Will not attempt to authenticate using SASL (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:16,202] INFO Socket error occurred: <FQDN>/<ip of
> localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:16,678] INFO [ZooKeeperClient] Closing.
> (kafka.zookeeper.ZooKeeperClient)
> [2020-11-19 22:33:17,303] INFO Opening socket connection to server <FQDN>/<ip
> of localhost>:5181. Will not attempt to authenticate using SASL (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:17,406] INFO Session: 0x0 closed
> (org.apache.zookeeper.ZooKeeper)
> [2020-11-19 22:33:17,407] INFO EventThread shut down for session: 0x0
> (org.apache.zookeeper.ClientCnxn)
> [2020-11-19 22:33:17,409] INFO [ZooKeeperClient] Closed.
> (kafka.zookeeper.ZooKeeperClient)
> [2020-11-19 22:33:17,412] ERROR Fatal error during KafkaServer startup.
> Prepare to shutdown (kafka.server.KafkaServer)
> kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
> connection while in state: CONNECTING
> at
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> at
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361)
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:205)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
> at kafka.Kafka$.main(Kafka.scala:75)
> at kafka.Kafka.main(Kafka.scala)
> [2020-11-19 22:33:17,415] INFO shutting down (kafka.server.KafkaServer)
> [2020-11-19 22:33:17,422] INFO shut down completed (kafka.server.KafkaServer)
> [2020-11-19 22:33:17,422] ERROR Exiting Kafka.
> (kafka.server.KafkaServerStartable)
> [2020-11-19 22:33:17,426] INFO shutting down (kafka.server.KafkaServer)
>
> 6. /var/log/messages
>
> Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None
> executable=None _uses_shell=False strip_empty_ends=True
> _raw_params=/opt/kafka/kafka_
> 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None
> argv=None warn=True chdir=None stdin_add_newline=True stdin=None
> Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None
> executable=None _uses_shell=False strip_empty_ends=True
> _raw_params=/opt/kafka/kafka_
> 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None
> argv=None warn=True chdir=None stdin_add_newline=True stdin=None
> Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None
> executable=None _uses_shell=False strip_empty_ends=True
> _raw_params=/opt/kafka/kafka_
> 2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None
> argv=None warn=True chdir=None stdin_add_newline=True stdin=None
> Nov 19 18:16:07 <HOSTNAME> heartbeat:
> 2020-11-19T18:16:07.941+0530#011INFO#011kafka/log.go:53#011client/metadata
> fetching metadata for [heartbeat] from bro
> ker <ip of localhost>:3997
> <ip of localhost>:3997: connect: connection refused
> 19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/metadata got error
> from broker -1 while fetching me
> tadata: dial tcp <ip of localhost>:3997: connect: connection refused
> Nov 19 18:16:07 <HOSTNAME> heartbeat:
> 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011kafka message:
> client/metadata no available broker to send
> metadata request to
> Nov 19 18:16:07 <HOSTNAME> heartbeat:
> 2020-11-19T18:16:07.942+0530#011INFO#011[publisher]#011pipeline/retry.go:155#011Drop
> batch
> Nov 19 18:16:07 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:07,962] INFO Reading configuration from:
> /opt/kafka/kafka_2.12-2.2.2/config/zookeep
> er.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,009] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of
> localhost> (org.apache.zookeeper.server.quorum.QuorumPeer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,009] ERROR Invalid configuration, only one server specified
> (ignoring) (org.apac
> he.zookeeper.server.quorum.QuorumPeerConfig)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,011] INFO autopurge.snapRetainCount set to 3
> (org.apache.zookeeper.server.Datadi
> rCleanupManager)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,011] INFO autopurge.purgeInterval set to 0
> (org.apache.zookeeper.server.DatadirC
> leanupManager)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,011] INFO Purge task is not scheduled.
> (org.apache.zookeeper.server.DatadirCleanupManager)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,011] WARN Either no config or no quorum defined in config, running
> in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain
>
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,027] INFO Reading configuration from:
> /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties
> (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,027] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of
> localhost> (org.apache.zookeeper.server.quorum.QuorumPeer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,027] ERROR Invalid configuration, only one server specified
> (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,027] INFO Starting server
> (org.apache.zookeeper.server.ZooKeeperServerMain)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,033] INFO Server
> environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03,
> built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,033] INFO Server environment:host.name=<FQDN>
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,033] INFO Server environment:java.version=1.8.0_221
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,033] INFO Server environment:java.vendor=Oracle Corporation
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,033] INFO Server environment:java.home=/opt/jre1.8.0_221
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,033] INFO Server
> environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.
12
>
> -2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar:
> 2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:java.compiler=<NA>
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:os.name=Linux
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:os.arch=amd64
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:os.version=3.10.0-1062.el7.x86_64
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:user.name=kafka
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:user.home=/home/kafka
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,034] INFO Server environment:user.dir=/
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,043] INFO tickTime set to 3000
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,043] INFO minSessionTimeout set to -1
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,044] INFO maxSessionTimeout set to -1
> (org.apache.zookeeper.server.ZooKeeperServer)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,049] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as
> server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,054] INFO binding to port 0.0.0.0/0.0.0.0:5181
> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19
> 18:16:08,131] ERROR Unexpected exception, exiting abnormally
> (org.apache.zookeeper.server.ZooKeeperServerMain)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: java.io.EOFException
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> java.io.DataInputStream.readInt(DataInputStream.java:392)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
> Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: main process exited,
> code=exited, status=1/FAILURE
> Nov 19 18:16:08 <HOSTNAME> zookeeper-server-stop.sh: No zookeeper server to
> stop
> Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: control process
> exited, code=exited status=1
> Nov 19 18:16:08 <HOSTNAME> systemd: Unit zookeeper.service entered failed
> state.
> Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service failed.
>
> 7. config file -
>
> **/opt/kafka/kafka_2.12-2.2.2/config/server.properties**
>
> broker.id=1
> advertised.listeners=PLAINTEXT://hostname of localhost:3997
> listeners=PLAINTEXT://hostname of localhost:3997
>
> delete.topic.enable=true
> auto.create.topics.enable=True
> offsets.topic.replication.factor=1
>
> # The number of threads handling network requests
> num.network.threads=3
>
> # The number of threads doing disk I/O
> num.io.threads=6
>
> # The send buffer (SO_SNDBUF) used by the socket server
> #socket.send.buffer.bytes=102400
>
> # The receive buffer (SO_RCVBUF) used by the socket server
> #socket.receive.buffer.bytes=102400
>
> # The maximum size of a request that the socket server will accept
> (protection against OOM)
> #socket.request.max.bytes=104857600
>
> # A comma seperated list of directories under which to store log files
> log.dirs=/var/log/kafka
>
> # The default number of log partitions per topic.
> num.partitions=3
>
> # Default Replication Factor
> default.replication.factor=1
>
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=168
>
> # A size-based retention policy for logs. Segments are pruned from the log as
> long as the remaining
> # segments don't drop below log.retention.bytes.
> #log.retention.bytes=1073741824
>
> # The maximum size of a log segment file. When this size is reached a new log
> segment will be created.
> log.segment.bytes=1073741824
>
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.retention.check.interval.ms=300000
>
> zookeeper.connect= <ip of localhost>:5181
> zookeeper.connection.timeout.ms=6000
>
>
> **/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties**
>
> # clientPortBindAddress=localhost
> clientPort=5181
>
> dataDir=/var/data/zookeeper
> dataLogDir=/var/data/logs/zookeeper
> maxClientCnxns=0
>
> # time for inital synchronization
> initLimit=10
> # how many ticks can pass before timeout
> syncLimit=5
>
> server.1=<hostname of localhost>:2888:3888
>
> 8.
>
> journalctl -fu zookeeper
> -- Logs begin at Thu 2020-11-19 01:05:17 IST. --
> Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
> Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
> Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
> Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
> Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
> Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
> Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: main process
> exited, code=exited, status=1/FAILURE
> Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: control process
> exited, code=exited status=1
> Nov 19 22:33:09 <HOSTNAME> systemd[1]: Unit zookeeper.service entered failed
> state.
> Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service failed.
>
> journalctl -fu kafka.service
> -- Logs begin at Thu 2020-11-19 01:05:17 IST. --
> Nov 19 22:23:30 <HOSTNAME> systemd[1]: Started kafka.service.
> Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: main process exited,
> code=exited, status=1/FAILURE
> Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: control process exited,
> code=exited status=1
> Nov 19 22:23:40 <HOSTNAME> systemd[1]: Unit kafka.service entered failed
> state.
> Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service failed.
> Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited,
> code=exited, status=1/FAILURE
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited,
> code=exited status=1
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed
> state.
> Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.
>
> this is the single server where everything is installed of elk along with
> tomcat etc.. so I have stopped other services like tomcat for better resource
> management. hope resource crunch is not the issue.
>
> free -g
> total used free shared buff/cache available
> Mem: 15 7 6 0 1 7
> Swap: 3 0 3
>
> Thanks,
>
