Greetings to the Kafka Community!
I'm a newbie in Kafka and only recently went beyond a local installation
described in the Quickstart. I have faced a weird issue that I can't
explain.
I want to deploy on two machines:
- Machine #1 runs ZooKeeper and a single Kafka broker. I use default
configuration files.
- Machine #2 runs a single instance of Kafka Connect. I use the source
connector for Ignite/GridGain (
https://www.confluent.io/hub/gridgain/ignite-connector). I doubt it
matters, though, as the issue seems to be related to connectivity between
the connector and the broker.
After starting the ZooKeeper and the broker on machine #1, I did the
following on machine #2:
1. Updated plugin.path and bootstrap.servers parameters in the
connect-standalone.properties; bootstrap.servers points to the IP of the #1.
2. Created a properties file for the GridGain connector.
3. Start the connector using this command:
bin/connect-standalone.sh config/connect-standalone.properties
config/gridgain-kafka-connect-source.properties
The issue is that it doesn't connect to the remote broker, and still tries
localhost ignoring the bootstrap.servers settings. The output looks like
this:
[2020-04-27 19:37:08,334] INFO StandaloneConfig values:
...
bootstrap.servers = [98.1.96.147:9092]
...
[2020-04-27 19:37:08,334] INFO Creating Kafka admin client
(org.apache.kafka.connect.util.ConnectUtils:43)
[2020-04-27 19:37:08,337] INFO AdminClientConfig values:
bootstrap.servers = [98.1.96.147:9092]
...
[2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1]
Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1]
Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1]
Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
The warning goes on for a while until the process stops after a timeout.
Does anyone have any idea why Kafka Connect can try to establish a
connection with the localhost, although it has picked up the correct IP
from the configuration file?
The issue doesn't seem to be reproducible on any environment, so I
would appreciate any pointers on how to investigate this further.
Kafka's version is 2.4.1. Config files and the log are attached. If
anything else is needed, let me know.
Regards,
Val
[2020-04-27 19:37:08,334] INFO StandaloneConfig values:
access.control.allow.methods =
access.control.allow.origin =
admin.listeners = null
bootstrap.servers = [98.1.96.147:9092]
client.dns.lookup = default
config.providers = []
connector.client.config.override.policy = None
header.converter = class
org.apache.kafka.connect.storage.SimpleHeaderConverter
internal.key.converter = class
org.apache.kafka.connect.json.JsonConverter
internal.value.converter = class
org.apache.kafka.connect.json.JsonConverter
key.converter = class org.apache.kafka.connect.json.JsonConverter
listeners = null
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
offset.flush.interval.ms = 10000
offset.flush.timeout.ms = 5000
offset.storage.file.filename = standalone.offsets
plugin.path = [/xxx/kafka_2.11-2.4.1/connect/gridgain-kafka-connect]
rest.advertised.host.name = null
rest.advertised.listener = null
rest.advertised.port = null
rest.extension.classes = []
rest.host.name = null
rest.port = 8083
ssl.client.auth = none
task.shutdown.graceful.timeout.ms = 5000
value.converter = class org.apache.kafka.connect.json.JsonConverter
(org.apache.kafka.connect.runtime.standalone.StandaloneConfig:347)
[2020-04-27 19:37:08,334] INFO Creating Kafka admin client
(org.apache.kafka.connect.util.ConnectUtils:43)
[2020-04-27 19:37:08,337] INFO AdminClientConfig values:
bootstrap.servers = [98.1.96.147:9092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 120000
retries = 5
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
(org.apache.kafka.clients.admin.AdminClientConfig:347)
[2020-04-27 19:37:08,382] WARN The configuration 'offset.flush.interval.ms' was
supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:355)
[2020-04-27 19:37:08,382] WARN The configuration 'key.converter.schemas.enable'
was supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:355)
[2020-04-27 19:37:08,383] WARN The configuration 'offset.storage.file.filename'
was supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:355)
[2020-04-27 19:37:08,383] WARN The configuration
'value.converter.schemas.enable' was supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:355)
[2020-04-27 19:37:08,383] WARN The configuration 'plugin.path' was supplied but
isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
[2020-04-27 19:37:08,383] WARN The configuration 'value.converter' was supplied
but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
[2020-04-27 19:37:08,383] WARN The configuration 'key.converter' was supplied
but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355)
[2020-04-27 19:37:08,383] INFO Kafka version: 2.4.1
(org.apache.kafka.common.utils.AppInfoParser:117)
[2020-04-27 19:37:08,383] INFO Kafka commitId: c57222ae8cd7866b
(org.apache.kafka.common.utils.AppInfoParser:118)
[2020-04-27 19:37:08,383] INFO Kafka startTimeMs: 1588030628383
(org.apache.kafka.common.utils.AppInfoParser:119)
[2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,555] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,957] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:10,860] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:11,964] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:12,867] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:13,870] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:15,073] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:16,177] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:17,079] WARN [AdminClient clientId=adminclient-1] Connection
to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient:756)
