I'll prefix this with I'm new to AWS. I'm used to Google's stack. I also noticed the helm instructions <https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes> mention kops (so I'm not sure if this works with EKS).
@Pirow - Kafka - I'm having issues setting up listeners and advertised.listeners. I added more details below. - Kubernetes - I've tried a few ways. Longer-term, I want it in the same VPC but outside of EKS. I tried the 3 ways listed in the helm instructions <https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes> (NodePort, Loadbalancer w/ distinct and Loadbalancer w/o distinct). The LoadBalancer with distinct route creates a load balancer with port <30000. I was relying on AWS's external IP for this. @Robin - Thanks! This link was very useful in understanding the problem. I'm still navigating the exact problem. What's the preferred way on AWS EKS? NodePort? When I used the NodePort helm instructions but I'm not sure what to put for the address. If I put in a single node in the cluster, e.g. ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com, the ports 31090-31092 seem to go to the same machine. I'm guessing I wouldn't want this to run in production since it's referring to a direct address for a machine. % diff kafka-values-original.yaml kafka-values-nodeport2.yaml 140c140 < enabled: false --- > enabled: true 154c154 < domain: cluster.local --- > domain: ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com 191,192c191,192 < # "advertised.listeners": |- < # EXTERNAL://kafka.cluster.local:$((31090 + ${KAFKA_BROKER_ID})) --- > "advertised.listeners": |- > EXTERNAL://ec2-35-170-61-153.compute-1.amazonaws.com:$((31090 + ${KAFKA_BROKER_ID})) 200,201c200,201 < # "listener.security.protocol.map": |- < # PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT --- > "listener.security.protocol.map": |- > PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT If I do this setup, the ports don't seem to go to the respective brokers. Which I'm guessing means that they're still talking to the same node. % kafkacat -b ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31091 -L Metadata for all topics (from broker -1: ec2-XXX-XXX-247-184.compute-1.amazonaws.com:31091/bootstrap): 3 brokers: broker 2 at ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31092 broker 1 at ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31091 broker 0 at ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31090 (controller) 0 topics: When I run my go code that uses segmentio/kafka-go, it seems to append an extra port: "dial tcp: address ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31090:31090: too many colons in address" On Thu, Mar 19, 2020 at 2:21 AM Robin Moffatt <ro...@confluent.io> wrote: > You need to make sure you've configured your listeners & advertised > listeners correctly. This should help: > https://rmoff.net/2018/08/02/kafka-listeners-explained/ > > > -- > > Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff > > > On Thu, 19 Mar 2020 at 01:49, Dan Hill <quietgol...@gmail.com> wrote: > > > Problem: I'm hitting an error: "no such host" for " > > kafka-0.cluster.local:19092". > > > > Has anyone done this before? Any help would be appreciated. Thanks! - > Dan > > > > My long-term goal is to get an AWS Lambda to send events to a Kafka > running > > in AWS EKS. > > > > I used the following instructions > > < > > > https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes > > > > > (linked to the "outside kubernetes" part) to setup up Kafka using the > helm > > config. The only modifications are for the "outside kubernetes" part. > > < > > > https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes > > > > > > > I've tried a few variations. None of them worked. I still can't connect > > to it. > > - on an Lambda in the same subnet, on an EC2 machine in the same subnet, > on > > a > > - with a couple different "outside kubernetes" options. > > > > E.g. if I setup external using LoadBalancer, I'll get something an > External > > IP like (fake) afdsafsafsafdas-13412341.us-east-1.elb.amazon.com:19092. > > > > If I try to run a basic command against this domain, it fails saying > there > > is no such host for kafka-0.cluster.local:19092. > > >
