Greetings to the Kafka Community! I'm a newbie in Kafka and only recently went beyond a local installation described in the Quickstart. I have faced a weird issue that I can't explain.
I want to deploy on two machines: - Machine #1 runs ZooKeeper and a single Kafka broker. I use default configuration files. - Machine #2 runs a single instance of Kafka Connect. I use the source connector for Ignite/GridGain ( https://www.confluent.io/hub/gridgain/ignite-connector). I doubt it matters, though, as the issue seems to be related to connectivity between the connector and the broker. After starting the ZooKeeper and the broker on machine #1, I did the following on machine #2: 1. Updated plugin.path and bootstrap.servers parameters in the connect-standalone.properties; bootstrap.servers points to the IP of the #1. 2. Created a properties file for the GridGain connector. 3. Start the connector using this command: bin/connect-standalone.sh config/connect-standalone.properties config/gridgain-kafka-connect-source.properties The issue is that it doesn't connect to the remote broker, and still tries localhost ignoring the bootstrap.servers settings. The output looks like this: [2020-04-27 19:37:08,334] INFO StandaloneConfig values: ... bootstrap.servers = [98.1.96.147:9092] ... [2020-04-27 19:37:08,334] INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils:43) [2020-04-27 19:37:08,337] INFO AdminClientConfig values: bootstrap.servers = [98.1.96.147:9092] ... [2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) The warning goes on for a while until the process stops after a timeout. Does anyone have any idea why Kafka Connect can try to establish a connection with the localhost, although it has picked up the correct IP from the configuration file? The issue doesn't seem to be reproducible on any environment, so I would appreciate any pointers on how to investigate this further. Kafka's version is 2.4.1. Config files and the log are attached. If anything else is needed, let me know. Regards, Val
[2020-04-27 19:37:08,334] INFO StandaloneConfig values: access.control.allow.methods = access.control.allow.origin = admin.listeners = null bootstrap.servers = [98.1.96.147:9092] client.dns.lookup = default config.providers = [] connector.client.config.override.policy = None header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter internal.key.converter = class org.apache.kafka.connect.json.JsonConverter internal.value.converter = class org.apache.kafka.connect.json.JsonConverter key.converter = class org.apache.kafka.connect.json.JsonConverter listeners = null metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 offset.flush.interval.ms = 10000 offset.flush.timeout.ms = 5000 offset.storage.file.filename = standalone.offsets plugin.path = [/xxx/kafka_2.11-2.4.1/connect/gridgain-kafka-connect] rest.advertised.host.name = null rest.advertised.listener = null rest.advertised.port = null rest.extension.classes = [] rest.host.name = null rest.port = 8083 ssl.client.auth = none task.shutdown.graceful.timeout.ms = 5000 value.converter = class org.apache.kafka.connect.json.JsonConverter (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:347) [2020-04-27 19:37:08,334] INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils:43) [2020-04-27 19:37:08,337] INFO AdminClientConfig values: bootstrap.servers = [98.1.96.147:9092] client.dns.lookup = default client.id = connections.max.idle.ms = 300000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 120000 retries = 5 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT security.providers = null send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS (org.apache.kafka.clients.admin.AdminClientConfig:347) [2020-04-27 19:37:08,382] WARN The configuration 'offset.flush.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) [2020-04-27 19:37:08,382] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) [2020-04-27 19:37:08,383] WARN The configuration 'offset.storage.file.filename' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) [2020-04-27 19:37:08,383] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) [2020-04-27 19:37:08,383] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) [2020-04-27 19:37:08,383] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) [2020-04-27 19:37:08,383] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) [2020-04-27 19:37:08,383] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser:117) [2020-04-27 19:37:08,383] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser:118) [2020-04-27 19:37:08,383] INFO Kafka startTimeMs: 1588030628383 (org.apache.kafka.common.utils.AppInfoParser:119) [2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:09,555] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:09,957] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:10,860] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:11,964] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:12,867] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:13,870] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:15,073] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:16,177] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756) [2020-04-27 19:37:17,079] WARN [AdminClient clientId=adminclient-1] Connection to node 0 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)