I am trying to configure a brand new Kafka cluster/sandbox with SSL, but I
keep getting errors. I apologize for the length of this email, but I've
never worked with keystores/certificates before, so while I'm trying to
follow the directions here (
http://kafka.apache.org/documentation.html#security_ssl), there are a few
things I'm doing my best on but don't quite understand. So I'm trying to
include below not just exactly what commands I'm running on exactly which
nodes, but my interpretation of exactly what they should be doing. I'm also
not trying to get client authentication working for the brokers yet - that
will be the next step. :-)
Setup: I have a brand new 3 node Kafka cluster, and I have 1 Edge node I am
going to use as a "CA-node".
Step 1:
I need to generate an SSL key and certificate for each Kafka broker.
On all 3 brokers:
keytool -keystore server.keystore.jks -alias localhost -validity 365
-genkey -keyalg RSA -storepass test1234
When it asks for first/last name, I am giving the FQDN of that broker.
I now have a keystore named server.keystore.jks on each broker. Later we
will export the certificate from this keystore so it can be signed by the CA
Step 2:
I need to create a CA, which is a public-private key pair plus a
certificate. We will use this CA to sign all 3 broker certificates, and as
long as all 3 brokers trust the CA, they will be able to trust each other
when they connect.
On the Edge Node that will be the CA:
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
For common name, I am using the FQDN of server
At this point I have a "ca-key" and "ca-cert" on the Edge node. Create a
"server.truststore.jks" and "client.truststore.jks" by executing the below:
keytool -keystore server.truststore.jks -alias CARoot -import -file
ca-cert
keytool -keystore client.truststore.jks -alias CARoot -import -file
ca-cert
At this point I have a "server.truststore.jks" and a
"client.truststore.jks" on the Edge Node, as well "ca-key" and "ca-cert"
Step 3:
Step 1 created a keystore on each machine. Step 2 created a CA on 1 machine
(Edge Node). Now I need to sign each certificate from Step 1 with the CA
from Step 2.
On all 3 brokers: Export the certificate from the keystore
keytool -keystore server.keystore.jks -alias localhost -certreq -file
cert-file
At this point each broker has a local "cert-file" (an exported
certificate). I need to sign those with the CA, using the ca-key and
ca-cert. But at this point, the ca-key and ca-cert are on the Edge Node/CA,
while the 3 individual certificates are on the 3 separate brokers. So, I am
going to (one node at a time) copy the certificates from the brokers onto
the CA node and run the command there (which seems better than copying a
private key around the cluster).
On the Edge/CA node (1 node at a time):
scp <username>@<FQDN of broker node>:/tmp/cert-file .
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out
cert-signed -days 365 -CAcreateserial -passin pass:test1234
At this point i now have a ca-cert.srl file and a cert-signed file on the
Edge/CA node. But the ca-cert (certificate for the CA) + cert-signed (that
I just generated) need to be on the broker nodes so I can import them into
the individual broker's keystores, so I am going to copy them back there:
On all 3 brokers (1 node at a time):
scp <username>@<FQDN of Edge/CA node>:/tmp/ca-cert .
scp <username>@<FQDN of Edge/CA node>:/tmp/cert-signed .
keytool -keystore server.keystore.jks -alias CARoot -import -file
ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file
cert-signed
>From checking the directions, it appears that each broker should have a
local file named "server.truststore.jks". But, I don't. Perhaps I followed
something too literally? I did create a server.truststore.jks file in Step
2, but it's sitting on the Edge/CA node. This doesn't seem right, so I am
going to instead execute the below command (from Step 2) on all 3 brokers.
The ca-cert file was already copied to the brokers so this should work.
keytool -keystore server.truststore.jks -alias CARoot -import -file
ca-cert
I then add the following lines to my server.properties file on each broker:
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.endpoint.identification.algorithm=HTTPS
ssl.key.password=test1234
ssl.keystore.location=/var/private/ssl/server.keystore.jks
ssl.keystore.password=test1234
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=test1234
listeners=SSL://<FQDN>:9093
security.inter.broker.protocol=SSL
When I start up Kafka after all of the changes above, I don't see any
errors in the logs...but I don't see very many logging entries at all. The
Kafka process is definitely up on all 3 brokers, but there's just not much
in the logs. When I run the suggested:
openssl s_client -debug -connect FQDN:9093 -tls1
I get something of the form:
<hexdump>
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
...
Verify return code: 19 (self signed certificate in certificate chain)
I'm not sure if that "self signed certificate" error means I did something
wrong above?
When I create the following properties file for my consumer/producer on my
Edge/CA node:
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/client.truststore.jks
ssl.truststore.password=test1234
and start up a producer/consumer quick test:
kafka-console-consumer.sh --bootstrap-server
broker1:9093,broker2:9093,broker3:9093 --topic withssl --consumer.config
ssl.properties
kafka-console-producer.sh --broker-list
broker1:9093,broker2:9093,broker3:9093 --topic withssl --producer.config
ssl.properties
Both give me the same error:
[2017-08-30 18:07:58,233] WARN Bootstrap broker broker0:9093 disconnected
(org.apache.kafka.clients.NetworkClient)
[2017-08-30 18:07:58,544] WARN Bootstrap broker broker1:9093 disconnected
(org.apache.kafka.clients.NetworkClient)
[2017-08-30 18:07:58,760] WARN Bootstrap broker broker2:9093 disconnected
(org.apache.kafka.clients.NetworkClient)
I'm not sure if my 3 brokers are set up incorrectly, or it my clients just
can't connect. Does anyone have any advice?
