A load balancer that balances the load across the brokers wouldn't work, but here the LB is being used as a terminating SSL proxy. That should work if each Kafka is configured with its own proxy.
On Mon, Nov 21, 2016 at 2:57 PM, tao xiao <xiaotao...@gmail.com> wrote: > I doubt the LB solution will work for Kafka. Client needs to connect to the > leader of a partition to produce/consume messages. If we put a LB in front > of all brokers which means all brokers share the same LB how does the LB > figure out the leader? > On Mon, Nov 21, 2016 at 10:26 PM Martin Gainty <mgai...@hotmail.com> > wrote: > > > > > > > > > > > ________________________________ > > From: Zac Harvey <zac.har...@welltok.com> > > Sent: Monday, November 21, 2016 8:59 AM > > To: users@kafka.apache.org > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > Thanks again Rajini, > > > > > > Using these configs, would clients connect to the load balancer over > > SSL/9093? And then would I configure the load balancer to forward traffic > > from SSL/9093 to plaintext/9093? > > > > MG>Zach > > > > MG>i could be wrong but SSL port != plaintext port ..but consider: > > > > MG>consider recent testcase where all traffic around a certain location > > gets bogged with DOS attacks > > > > MG>what are the legitimate role(s) of the LB when SSL Traffic and HTTP1.1 > > Traffic and FTP Traffic are ALL blocked? > > > > MG>LB should never be stripping SSL headers to redirect to PlainText > > because you are not rerouting to a faster route > > > > MG>most net engineers worth their salt will configure their routers to > > static routes to loop around bogged-down routers > > > > MG>WDYT? > > > > Thanks again, just still a little uncertain about the traffic/ports > coming > > into the load balancer! > > > > > > Best, > > > > Zac > > > > ________________________________ > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > Sent: Monday, November 21, 2016 8:48:41 AM > > To: users@kafka.apache.org > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > Zac, > > > > Yes, that is correct. Ruby clients will not be authenticated by Kafka. > They > > talk SSL to the load balancer and the load balancer uses PLAINTEXT > without > > authentication to talk to Kafka. > > > > On Mon, Nov 21, 2016 at 1:29 PM, Zac Harvey <zac.har...@welltok.com> > > wrote: > > > > > *Awesome* explanation Rajini - thank you! > > > > > > > > > Just to confirm: the SASL/PLAIN configs would only be for the > interbroker > > > communication, correct? Meaning, beyond your recommended changes to > > > server.properties, and the addition of the new jaas.conf file, the > > > producers (Ruby clients) wouldn't need to authenticate, correct? > > > > > > > > > Thanks again for all the great help so far, you've already helped me > more > > > than you know! > > > > > > > > > Zac > > > > > > ________________________________ > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > > Sent: Monday, November 21, 2016 3:53:47 AM > > > To: users@kafka.apache.org > > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > > > Zac, > > > > > > *advertised.listeners* is used to make client connections from > > > producers/consumers as well as for client-side connections for > > inter-broker > > > communication. In your scenario, setting it to *PLAINTEXT://mykafka01* > > > would work for inter-broker, bypassing the load balancer, but clients > > would > > > also then attempt to connect directly to *mykafka01*. Setting it to > > > *SSL://mybalancer01* would work for producers/consumers, but brokers > > would > > > try to connect to *mybalancer01* using PLAINTEXT. Unfortunately neither > > > works for both. You need two endpoints, one for inter-broker that > > bypasses > > > *mybalancer01* and another for clients that uses *mybalancer01*. With > the > > > current Kafka configuration, you would require two security protocols > to > > > enable two endpoints. > > > > > > You could enable SSL in Kafka (using self-signed certificates if you > > need) > > > for one of the two endpoints to overcome this limitation. But > presumably > > > you have a secure internal network running Kafka and want to avoid the > > cost > > > of encryption in Kafka. The simplest solution I can think of is to use > > > SASL_PLAINTEXT using SASL/PLAIN for inter-broker as a workaround. The > > > configuration options in server.properties would look like: > > > > > > listeners=PLAINTEXT://:9093,SASL_PLAINTEXT://:9092 > > > > > > advertised.listeners=PLAINTEXT://mybalancer01.example.com:9093 > > > ,SASL_PLAINTEXT://mykafka01.example.com:9092 > > > > > > security.inter.broker.protocol=SASL_PLAINTEXT > > > > > > sasl.enabled.mechanisms=PLAIN > > > > > > sasl.mechanism.inter.broker.protocol=PLAIN > > > > > > > > > You also need a JAAS configuration file configured for the broker JVM ( > > > *KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/jaas.conf"*) . > See > > > https://kafka.apache.org/documentation#security_sasl for configuring > > > SASL.* > > > jaas.conf* would look something like: > > > > > > KafkaServer { > > > > > > org.apache.kafka.common.security.plain.PlainLoginModule > required > > > > > > username="kafka" > > > > > > user_kafka="kafka-password" > > > > > > password="kafka-password"; > > > > > > }; > > > > > > > > > Hope that helps. > > > > > > > > > On Fri, Nov 18, 2016 at 6:39 PM, Zac Harvey <zac.har...@welltok.com> > > > wrote: > > > > > > > Thanks again Rajini! > > > > > > > > > > > > One last followup question, if you don't mind. You said that my > > > > server.properties file should look something like this: > > > > > > > > > > > > listeners=SSL://:9093 > > > > advertised.listeners=SSL://mybalancer01.example.com:9093 > > > > security.inter.broker.protocol=SSL > > > > > > > > However, please remember that I'm looking for the load balancer to > > > > terminate SSL, meaning that (my desired) communication between the > load > > > > balancer and Kafka would be over plaintext (not SSL). In other > words: > > > > > > > > Ruby Producers/Clients <----SSL:9093----> Load Balancer <---- > > > > Plaintext:9092 ----> Kafka > > > > > > > > So producers/client connect to the load balancer over SSL and port > > 9093, > > > > but then the load balancer communicates with Kafka over plaintext and > > > port > > > > 9092. > > > > > > > > I also don't need inter broker communication to be SSL; it can be > > > > plaintext. > > > > > > > > If this is the case, do I still need to change server.properties, or > > can > > > I > > > > leave it like so: > > > > > > > > listeners=plaintext://:9092 > > > > advertised.listeners=plaintext://mybalancer01.example.com:9092 > > > > > > > > Or could it just be: > > > > > > > > listeners=plaintext://:9092 > > > > advertised.listeners=plaintext://mykafka01.example.com:9092 > > > > > > > > Thanks again! > > > > Zac > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > > > Sent: Friday, November 18, 2016 9:57:22 AM > > > > To: users@kafka.apache.org > > > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > > > > > You should set advertised.listeners rather than the older > > > > advertised.host.name property in server.properties: > > > > > > > > > > > > - listeners=SSL://:9093 > > > > - advertised.listeners=SSL://mybalancer01.example.com:9093 > > > > - security.inter.broker.protocol=SSL > > > > > > > > > > > > If your listeners are on particular interfaces, you can set address > in > > > the > > > > 'listeners' property too. > > > > > > > > > > > > If you want inter-broker communication to bypass the SSL proxy, you > > would > > > > need another security protocol that can be used for inter-broker > > > > communication (PLAINTEXT in the example below). > > > > > > > > > > > > > > > > - listeners=SSL://:9093,PLAINTEXT://:9092 > > > > - advertised.listeners=SSL://mybalancer01.example.com:9093, > > > PLAINTEXT:// > > > > mykafka01.example.com:9092 > > > > - security.inter.broker.protocol=PLAINTEXT > > > > > > > > I haven't used the Ruby clients, so I am not sure about client > > > > configuration. With Java clients, if you don't specify truststore, > the > > > > default trust stores are used, so with trusted CA-signed > certificates, > > no > > > > additional client configuration is required. You can test your > > > installation > > > > using the console producer and consumer that are shipped with Kafka > to > > > make > > > > sure it is working before you run with Ruby clients. > > > > > > > > > > > > > > > > On Fri, Nov 18, 2016 at 1:23 PM, Zac Harvey <zac.har...@welltok.com> > > > > wrote: > > > > > > > > > > > > > > Thanks Rajini, > > > > > > > > > > > > > > > So currently one of our Kafka nodes is 'mykafka01.example.com', > and > > in > > > > > its server.properties file, I have advertised.host.name=mykafka01 > > > > > .example.com. Our load balancer lives at mybalancer01.example.com, > > and > > > > > this what producers will connect to (over SSL) to send messages to > > > Kafka. > > > > > > > > > > > > > > > It sounds like you're saying I need to change my Kafka node's > > > > > server.properties to have advertised.host.name=mybalance > > > r01.example.com, > > > > > yes? If not, can you perhaps provide a quick snippet of the > changes I > > > > would > > > > > need to make to server.properties? > > > > > > > > > > > > > > > Again, the cert served by the balancer will be a highly-trusted > (root > > > > > CA-signed) certificate that all clients will natively trust. > > > > Interestingly > > > > > enough, most (if not all) the Kafka producers/clients will be > written > > > in > > > > > Ruby (using the zendesk Kafka-Ruby gem<https://github.com/ > > > > > zendesk/ruby-kafka>), so there wont be any JKS configuration > options > > > > > available for those Ruby clients. > > > > > > > > > > > > > > > Besides making the change to server.properties that I mentioned > > above, > > > > are > > > > > there any other client-side configs that will need to be made for > the > > > > Ruby > > > > > clients to connect over SSL? > > > > > > > > > > > > > > > Thank you enormously here! > > > > > > > > > > > > > > > Best, > > > > > > > > > > Zac > > > > > > > > > > > > > > > ________________________________ > > > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > > > > Sent: Friday, November 18, 2016 5:15:13 AM > > > > > To: users@kafka.apache.org > > > > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > > > > > > > Zac, > > > > > > > > > > Kafka has its own built-in load-balancing mechanism based on > > partition > > > > > assignment. Requests are processed by partition leaders, > distributing > > > > load > > > > > across the brokers in the cluster. If you want to put a proxy like > > > > HAProxy > > > > > with SSL termination in front of your brokers for added security, > you > > > can > > > > > do that. You can have completely independent trust chain between > > > > > clients->proxy and proxy->broker. You need to configure Kafka > brokers > > > > with > > > > > the proxy host as the host in the advertised listeners for the > > security > > > > > protocol used by clients. > > > > > > > > > > On Thu, Nov 17, 2016 at 9:44 PM, Zac Harvey < > zac.har...@welltok.com> > > > > > wrote: > > > > > > > > > > > We have two Kafka nodes and for reasons outside of this question, > > > would > > > > > > like to set up a load balancer to terminate SSL with producers > > > > (clients). > > > > > > The SSL cert hosted by the load balancer will be signed by > > > trusted/root > > > > > CA > > > > > > that clients should natively trust. > > > > > > > > > > > > > > > > > > Is this possible to do, or does Kafka somehow require SSL to be > > setup > > > > > > directly on the Kafka servers themselves? > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Rajini > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Rajini > > > > > > > > > > > -- > > Regards, > > > > Rajini > > > -- Regards, Rajini
