Ah, I think I figured out part of my issue. If I update the Helm chart values that impact advertised.listeners and thenI run a 'helm upgrade', it sometimes does not actually apply the settings immediately. This made debugging really hard until I figured this out. I don't know Helm/Kubernetes well enough but this seems wrong/bad. The result of 'upgrade' varied depending on the fields I updated. I assumed 'helm upgrade' should get the servers to the state in the configs. Once I realized this and switched to uninstalling and then installing, I was able to get to a working configuration.
On Thu, Mar 19, 2020 at 4:55 PM Dan Hill <quietgol...@gmail.com> wrote: > I'll prefix this with I'm new to AWS. I'm used to Google's stack. I also > noticed the helm instructions > <https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes> > mention > kops (so I'm not sure if this works with EKS). > > @Pirow > - Kafka - I'm having issues setting up listeners and > advertised.listeners. I added more details below. > - Kubernetes - I've tried a few ways. Longer-term, I want it in the same > VPC but outside of EKS. I tried the 3 ways listed in the helm > instructions > <https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes> > (NodePort, > Loadbalancer w/ distinct and Loadbalancer w/o distinct). The LoadBalancer > with distinct route creates a load balancer with port <30000. I was > relying on AWS's external IP for this. > > @Robin - Thanks! This link was very useful in understanding the problem. > I'm still navigating the exact problem. > > > > What's the preferred way on AWS EKS? NodePort? When I used the NodePort > helm instructions but I'm not sure what to put for the address. If I put > in a single node in the cluster, e.g. > ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com, the ports 31090-31092 seem > to go to the same machine. I'm guessing I wouldn't want this to run in > production since it's referring to a direct address for a machine. > > > % diff kafka-values-original.yaml kafka-values-nodeport2.yaml > 140c140 > < enabled: false > --- > > enabled: true > 154c154 > < domain: cluster.local > --- > > domain: ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com > 191,192c191,192 > < # "advertised.listeners": |- > < # EXTERNAL://kafka.cluster.local:$((31090 + ${KAFKA_BROKER_ID})) > --- > > "advertised.listeners": |- > > EXTERNAL://ec2-35-170-61-153.compute-1.amazonaws.com:$((31090 + > ${KAFKA_BROKER_ID})) > 200,201c200,201 > < # "listener.security.protocol.map": |- > < # PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT > --- > > "listener.security.protocol.map": |- > > PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT > > If I do this setup, the ports don't seem to go to the respective brokers. > Which I'm guessing means that they're still talking to the same node. > > % kafkacat -b ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31091 -L > Metadata for all topics (from broker -1: > ec2-XXX-XXX-247-184.compute-1.amazonaws.com:31091/bootstrap): > 3 brokers: > broker 2 at ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31092 > broker 1 at ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31091 > broker 0 at ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31090 > (controller) > 0 topics: > > When I run my go code that uses segmentio/kafka-go, it seems to append an > extra port: > "dial tcp: address ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:31090:31090: > too many colons in address" > > > > > On Thu, Mar 19, 2020 at 2:21 AM Robin Moffatt <ro...@confluent.io> wrote: > >> You need to make sure you've configured your listeners & advertised >> listeners correctly. This should help: >> https://rmoff.net/2018/08/02/kafka-listeners-explained/ >> >> >> -- >> >> Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff >> >> >> On Thu, 19 Mar 2020 at 01:49, Dan Hill <quietgol...@gmail.com> wrote: >> >> > Problem: I'm hitting an error: "no such host" for " >> > kafka-0.cluster.local:19092". >> > >> > Has anyone done this before? Any help would be appreciated. Thanks! - >> Dan >> > >> > My long-term goal is to get an AWS Lambda to send events to a Kafka >> running >> > in AWS EKS. >> > >> > I used the following instructions >> > < >> > >> https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes >> > > >> > (linked to the "outside kubernetes" part) to setup up Kafka using the >> helm >> > config. The only modifications are for the "outside kubernetes" part. >> > < >> > >> https://github.com/helm/charts/tree/master/incubator/kafka#connecting-to-kafka-from-outside-kubernetes >> > > >> > >> > I've tried a few variations. None of them worked. I still can't >> connect >> > to it. >> > - on an Lambda in the same subnet, on an EC2 machine in the same >> subnet, on >> > a >> > - with a couple different "outside kubernetes" options. >> > >> > E.g. if I setup external using LoadBalancer, I'll get something an >> External >> > IP like (fake) afdsafsafsafdas-13412341.us-east-1.elb.amazon.com:19092. >> > >> > If I try to run a basic command against this domain, it fails saying >> there >> > is no such host for kafka-0.cluster.local:19092. >> > >> >
