Hi Team,
I am running ansible to install elk on one server. (on Dev env, standalone
setup) but my playbook is getting failed at kafka role for task listing of
Kafka topics. Can you please help.
below are the logs.
1.
playbook getting failed with error: -
```
TASK [kafka : Kafka Topics | List topics]
******************************************************************************************************************************
fatal: [kafka_1]: FAILED! => {
"changed": false,
"cmd": [
"/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh",
"--zookeeper",
"<FQDN>:5181",
"--list"
],
"delta": "0:00:34.084478",
"end": "2020-11-19 22:33:42.624353",
"rc": 1,
"start": "2020-11-19 22:33:08.539875"
}
STDERR:
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException:
Timed out waiting for connection while in state: CONNECTING
at
kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at
kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
```
2. Getting same error when run command manually which playbook is trying to
run.
```
/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper <ip of
localhost>:5181 --list
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException:
Timed out waiting for connection while in state: CONNECTING
at
kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at
kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
at
kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
```
3. both Kafka and zookeeper are failed and not getting started.
```
systemctl status kafka.service -l
● kafka.service
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset:
disabled)
Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 IST; 5min
ago
Process: 5448 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh
(code=exited, status=1/FAILURE)
Process: 4742 ExecStart=/bin/sh -c
/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh
/opt/kafka/kafka_2.12-2.2.2/config/server.properties >
/opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, status=1/FAILURE)
Main PID: 4742 (code=exited, status=1/FAILURE)
Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited,
code=exited, status=1/FAILURE
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited,
code=exited status=1
Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.
systemctl status zookeeper.service -l
● zookeeper.service
Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor
preset: disabled)
Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 IST; 5min
ago
Process: 5281
ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh (code=exited,
status=1/FAILURE)
Process: 4448
ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh
/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (code=exited,
status=1/FAILURE)
Main PID: 4448 (code=exited, status=1/FAILURE)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 22:33:09 systemd[1]: zookeeper.service: main process exited,
code=exited, status=1/FAILURE
Nov 19 22:33:09 systemd[1]: zookeeper.service: control process exited,
code=exited status=1
Nov 19 22:33:09 systemd[1]: Unit zookeeper.service entered failed state.
Nov 19 22:33:09 systemd[1]: zookeeper.service failed.
```
4. therefore unable to do telnet or netstat
telnet <ip og localhost> 5181
Trying 10..
telnet: connect to address <ip of localhost>: Connection refused
netstat -tnlp |grep 5181
No output
No Iptables or SELinux is running.
5. output of /opt/kafka/kafka_2.12-2.2.2/kafka.log
[2020-11-19 22:33:09,853] INFO Registered kafka:type=kafka.Log4jController
MBean (kafka.utils.Log4jControllerRegistration$)
[2020-11-19 22:33:10,584] INFO starting (kafka.server.KafkaServer)
[2020-11-19 22:33:10,585] INFO Connecting to zookeeper on <ip of
localhost>:5181 (kafka.server.KafkaServer)
[2020-11-19 22:33:10,640] INFO [ZooKeeperClient] Initializing a new session to
<ip of localhost>:5181. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:10,646] INFO Client
environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03,
built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:host.name=<FQDN>
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.version=1.8.0_221
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.home=/opt/jre1.8.0_221
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client
environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-api-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-basic-auth-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-file-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-json-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-runtime-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/ka
fka_2.12-2.2.2/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-core-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-databind-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/..
/libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-http-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-io-9.4.14.v20181114.jar:/opt/kafk
a/kafka_2.12-2.2.2/bin/../libs/jetty-security-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-server-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlet-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-servlets-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-examples-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafka_2.
12-2.2.2/bin/../libs/kafka-tools-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/log4j-1.2.17.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/lz4-java-1.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/maven-artifact-3.6.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/reflections-0.9.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/rocksdbjni-5.15.10.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0
.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client
environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.name=kafka
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.home=/home/kafka
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.dir=/
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,648] INFO Initiating client connection, connectString=<ip
of localhost>:5181 sessionTimeout=6000
watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3c419631
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,667] INFO Opening socket connection to server <FQDN>/<ip
of localhost>:5181. Will not attempt to authenticate using SASL (unknown error)
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:10,673] INFO Socket error occurred: <FQDN>/<ip of
localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:10,676] INFO [ZooKeeperClient] Waiting until connected.
(kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:11,796] INFO Opening socket connection to server <FQDN>/<ip
of localhost>:5181. Will not attempt to authenticate using SASL (unknown error)
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:11,797] INFO Socket error occurred: <FQDN>/<ip of
localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:12,897] INFO Opening socket connection to server <FQDN>/<ip
of localhost>:5181. Will not attempt to authenticate using SASL (unknown error)
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:12,898] INFO Socket error occurred: <FQDN>/<ip of
localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:13,999] INFO Opening socket connection to server <FQDN>/<ip
of localhost>:5181. Will not attempt to authenticate using SASL (unknown error)
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:13,999] INFO Socket error occurred: <FQDN>/<ip of
localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:15,101] INFO Opening socket connection to server <FQDN>/<ip
of localhost>:5181. Will not attempt to authenticate using SASL (unknown error)
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:15,101] INFO Socket error occurred: <FQDN>/<ip of
localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,202] INFO Opening socket connection to server <FQDN>/<ip
of localhost>:5181. Will not attempt to authenticate using SASL (unknown error)
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,202] INFO Socket error occurred: <FQDN>/<ip of
localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,678] INFO [ZooKeeperClient] Closing.
(kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:17,303] INFO Opening socket connection to server <FQDN>/<ip
of localhost>:5181. Will not attempt to authenticate using SASL (unknown error)
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:17,406] INFO Session: 0x0 closed
(org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:17,407] INFO EventThread shut down for session: 0x0
(org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:17,409] INFO [ZooKeeperClient] Closed.
(kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:17,412] ERROR Fatal error during KafkaServer startup. Prepare
to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
connection while in state: CONNECTING
at
kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at
kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)
at kafka.server.KafkaServer.startup(KafkaServer.scala:205)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2020-11-19 22:33:17,415] INFO shutting down (kafka.server.KafkaServer)
[2020-11-19 22:33:17,422] INFO shut down completed (kafka.server.KafkaServer)
[2020-11-19 22:33:17,422] ERROR Exiting Kafka.
(kafka.server.KafkaServerStartable)
[2020-11-19 22:33:17,426] INFO shutting down (kafka.server.KafkaServer)
6. /var/log/messages
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None
executable=None _uses_shell=False strip_empty_ends=True
_raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None
argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None
executable=None _uses_shell=False strip_empty_ends=True
_raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None
argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None
executable=None _uses_shell=False strip_empty_ends=True
_raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None
argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> heartbeat:
2020-11-19T18:16:07.941+0530#011INFO#011kafka/log.go:53#011client/metadata
fetching metadata for [heartbeat] from bro
ker <ip of localhost>:3997
<ip of localhost>:3997: connect: connection refused
19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/metadata got error
from broker -1 while fetching me
tadata: dial tcp <ip of localhost>:3997: connect: connection refused
Nov 19 18:16:07 <HOSTNAME> heartbeat:
2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011kafka message:
client/metadata no available broker to send
metadata request to
Nov 19 18:16:07 <HOSTNAME> heartbeat:
2020-11-19T18:16:07.942+0530#011INFO#011[publisher]#011pipeline/retry.go:155#011Drop
batch
Nov 19 18:16:07 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:07,962]
INFO Reading configuration from: /opt/kafka/kafka_2.12-2.2.2/config/zookeep
er.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009]
INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost>
(org.apache.zookeeper.server.quorum.QuorumPeer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009]
ERROR Invalid configuration, only one server specified (ignoring) (org.apac
he.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011]
INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.Datadi
rCleanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011]
INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirC
leanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011]
INFO Purge task is not scheduled.
(org.apache.zookeeper.server.DatadirCleanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011]
WARN Either no config or no quorum defined in config, running in standalone
mode (org.apache.zookeeper.server.quorum.QuorumPeerMain
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027]
INFO Reading configuration from:
/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027]
INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost>
(org.apache.zookeeper.server.quorum.QuorumPeer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027]
ERROR Invalid configuration, only one server specified (ignoring)
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027]
INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033]
INFO Server
environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03,
built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033]
INFO Server environment:host.name=<FQDN>
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033]
INFO Server environment:java.version=1.8.0_221
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033]
INFO Server environment:java.vendor=Oracle Corporation
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033]
INFO Server environment:java.home=/opt/jre1.8.0_221
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033]
INFO Server
environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-2.3.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12
-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka_2.12-2.2.2-sources.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-clients-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-log4j-appender-2.2.2.jar:
2.2.2/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:java.compiler=<NA>
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:os.name=Linux
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:os.arch=amd64
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:os.version=3.10.0-1062.el7.x86_64
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:user.name=kafka
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:user.home=/home/kafka
(org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034]
INFO Server environment:user.dir=/ (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043]
INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043]
INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,044]
INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,049]
INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,054]
INFO binding to port 0.0.0.0/0.0.0.0:5181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,131]
ERROR Unexpected exception, exiting abnormally
(org.apache.zookeeper.server.ZooKeeperServerMain)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: java.io.EOFException
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
java.io.DataInputStream.readInt(DataInputStream.java:392)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: main process exited,
code=exited, status=1/FAILURE
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-stop.sh: No zookeeper server to stop
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: control process exited,
code=exited status=1
Nov 19 18:16:08 <HOSTNAME> systemd: Unit zookeeper.service entered failed state.
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service failed.
7. config file -
**/opt/kafka/kafka_2.12-2.2.2/config/server.properties**
broker.id=1
advertised.listeners=PLAINTEXT://hostname of localhost:3997
listeners=PLAINTEXT://hostname of localhost:3997
delete.topic.enable=true
auto.create.topics.enable=True
offsets.topic.replication.factor=1
# The number of threads handling network requests
num.network.threads=3
# The number of threads doing disk I/O
num.io.threads=6
# The send buffer (SO_SNDBUF) used by the socket server
#socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
#socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection
against OOM)
#socket.request.max.bytes=104857600
# A comma seperated list of directories under which to store log files
log.dirs=/var/log/kafka
# The default number of log partitions per topic.
num.partitions=3
# Default Replication Factor
default.replication.factor=1
# The minimum age of a log file to be eligible for deletion
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log as
long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log
segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted
according
# to the retention policies
log.retention.check.interval.ms=300000
zookeeper.connect= <ip of localhost>:5181
zookeeper.connection.timeout.ms=6000
**/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties**
# clientPortBindAddress=localhost
clientPort=5181
dataDir=/var/data/zookeeper
dataLogDir=/var/data/logs/zookeeper
maxClientCnxns=0
# time for inital synchronization
initLimit=10
# how many ticks can pass before timeout
syncLimit=5
server.1=<hostname of localhost>:2888:3888
8.
journalctl -fu zookeeper
-- Logs begin at Thu 2020-11-19 01:05:17 IST. --
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: main process exited,
code=exited, status=1/FAILURE
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: control process
exited, code=exited status=1
Nov 19 22:33:09 <HOSTNAME> systemd[1]: Unit zookeeper.service entered failed
state.
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service failed.
journalctl -fu kafka.service
-- Logs begin at Thu 2020-11-19 01:05:17 IST. --
Nov 19 22:23:30 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: main process exited,
code=exited, status=1/FAILURE
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: control process exited,
code=exited status=1
Nov 19 22:23:40 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service failed.
Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited,
code=exited, status=1/FAILURE
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited,
code=exited status=1
Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.
this is the single server where everything is installed of elk along with
tomcat etc.. so I have stopped other services like tomcat for better resource
management. hope resource crunch is not the issue.
free -g
total used free shared buff/cache available
Mem: 15 7 6 0 1 7
Swap: 3 0 3
Thanks,
