Hi! First time writing to this list, so excuse me if I am not following some conventions 😉
I am developing software using Kafka since about two years and normally I run the whole setup on my laptop including running Kafka in Docker to be able to test easily and this has been working fine. Now, however last week I had some trouble where I had to make a hard reboot of my laptop (a Macbook Pro) and after that I have had strange problems. Kafka starts up in the container nicely without errors but I cannot connect to it! The client can connect initially and then the connection fails with different errors depending on the client I use. I know, it’s a classic problem superficially and I have read the confluent article (https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/) and ran the client etc… The error python_kafka_test_client.py gives me is: %6|1616487438.952|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Disconnected (after 0ms in state APIVERSION_QUERY) Message received: "foo / 2021-03-23 09:17:17" from topic test_topic However the kicker here is that this all worked great before, before my reboot, with the same clients and the same settings. Additionally, I have re-installed Docker and I even ran Kafka outside Docker (using brew) and EVEN started Kafka on a different (private) laptop and I get the SAME errors when trying to connect to it as well! (yes I verified that I CAN connect nicely to that Kafka locally on the other laptop) Here are some sample log lines from one of our Spring Boot apps trying to connect to my local Kafka: 2021-03-22 18:51:55.926 [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=persister-v2-1, groupId=persister-v2] Error while fetching metadata with correlation id 2 : {odp-dml-persister-topic-v2=LEADER_NOT_AVAILABLE} 2021-03-22 18:51:55.926 [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=persister-v2-1, groupId=persister-v2] Cluster ID: qfvGlr0JRNC-JrBrONM-vQ 2021-03-22 18:51:55.956 [org.springframework.kafka.KafkaListenerEndpointContainer#0-4-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=persister-v2-4, groupId=persister-v2] Error while fetching metadata with correlation id 2 : {odp-dml-persister-topic-v2=LEADER_NOT_AVAILABLE} 2021-03-22 18:51:55.956 [org.springframework.kafka.KafkaListenerEndpointContainer#0-4-C-1] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=persister-v2-4, groupId=persister-v2] Cluster ID: qfvGlr0JRNC-JrBrONM-vQ Also, some sample errors from other clients I tried with trial versions etc: KaDeck 3.1.2 – Får ”Could not complete ListTopics action: connection could not be established. Timeout exception: Timed out waiting for node assignment” Conductor – “The broker [localhost: 9092] is reachable but Kafka can’t connect. Ensure you have access to the *advertised listeners* of the brokers and the proper authorizations” The odd thing is that ALL the other services that I start in Docker I can connect nicely too (Cassandra, Mysql, MongoDb etc)?!? I suspect this is something related to my network locally on my laptop that has become messed up/corrupted? So technically this is not a Kafka issue in itself, but I need help/tips on how to troubleshoot and get to the root of this. I even tried using Wireshark to look into the traffic, but its hard to see anything useful since the traffic is binary and I am neither a Wireshark expert nor expert on Kafka protocol (obviously). I have tried turning off my firewall and also did several reboots and re-installs of software etc. Can anyone help point me in a useful direction to solve this? Any tips on how to trace and figure out exactly where things goes wrong? I am really dependent on using Kafka locally for testing and I really am a bit stuck. My next desperate step could be to just wipe and re-install my laptop, but that could make me unproductive for several days… ☹ Best regards, Kent Närling
