Hi, May I ask for some help about setting up SSL for Zookeeper, please?
My goal is just to add SSL for my zookeeper server then just test it using openssl s_client. So far I got this from the logs (WARN) "Empty server certificate chain". So here's what I've done so far. #1 Install Kafka and Zookeeper in my Ubuntu machine (using multipass). I downloaded the latest version here <https://www.apache.org/dyn/closer.cgi?path=/kafka/2.8.0/kafka_2.12-2.8.0.tgz> #2 I was able to connect Kafka to Zookeeper just fine. Zookeeper: kafka@zoo:/etc/kafka$ bin/zookeeper-server-start.sh config/zookeeper.properties [image: image.png] #3 Test the SSL connection without setting up the zookeeper.properties and .jks files kafka@zoo:/etc/kafka$ openssl s_client -showcerts -connect localhost:2181 CONNECTED(00000003) write:errno=0 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 283 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- #4 Setup SSL zookeeper.properties and .jks files Below and the commands I've done to generate the and sign the SSL (the .jks files) Example steps to implement CA, truststore and keystore for zookeeper SSL security : 𝟏. 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐂𝐀 == openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650 𝟐. 𝐂𝐫𝐞𝐚𝐭𝐞 𝐓𝐫𝐮𝐬𝐭𝐬𝐭𝐨𝐫𝐞 == keytool -keystore kafka.zookeeper.truststore.jks -alias ca-cert -import -file ca-cert 𝟑. 𝐂𝐫𝐞𝐚𝐭𝐞 𝐊𝐞𝐲𝐬𝐭𝐨𝐫𝐞 == keytool -keystore kafka.zookeeper.keystore.jks -alias zookeeper -validity 3650 -genkey -keyalg RSA -ext SAN=dns:localhost 𝟒. 𝐂𝐫𝐞𝐚𝐭𝐞 𝐜𝐞𝐫𝐭𝐢𝐟𝐢𝐜𝐚𝐭𝐞 𝐬𝐢𝐠𝐧𝐢𝐧𝐠 𝐫𝐞𝐪𝐮𝐞𝐬𝐭 (𝐂𝐒𝐑) == keytool -keystore kafka.zookeeper.keystore.jks -alias zookeeper -certreq -file ca-request-zookeeper 𝟓. 𝐒𝐢𝐠𝐧 𝐭𝐡𝐞 𝐂𝐒𝐑 == openssl x509 -req -CA ca-cert -CAkey ca-key -in ca-request-zookeeper -out ca-signed-zookeeper -days 3650 -CAcreateserial 𝟔. 𝐈𝐦𝐩𝐨𝐫𝐭 𝐭𝐡𝐞 𝐂𝐀 𝐢𝐧𝐭𝐨 𝐊𝐞𝐲𝐬𝐭𝐨𝐫𝐞 == keytool -keystore kafka.zookeeper.keystore.jks -alias ca-cert -import -file ca-cert 𝟕. 𝐈𝐦𝐩𝐨𝐫𝐭 𝐭𝐡𝐞 𝐬𝐢𝐠𝐧𝐞𝐝 𝐜𝐞𝐫𝐭𝐢𝐟𝐢𝐜𝐚𝐭𝐞 𝐢𝐧𝐭𝐨 𝐊𝐞𝐲𝐬𝐭𝐨𝐫𝐞 == keytool -keystore kafka.zookeeper.keystore.jks -alias zookeeper -import -file ca-signed-zookeeper See the files are there in /etc/kafka kafka@zoo:/etc/kafka$ ls LICENSE bin ca-cert.srl ca-request-zookeeper config kafka.zookeeper.truststore.jks licenses site-docs NOTICE ca-cert ca-key ca-signed-zookeeper kafka.zookeeper.keystore.jks libs logs I added some lines in zookeeper.properties and here's the content of that file: kafka@zoo:/etc/kafka$ cat config/zookeeper.properties # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # the directory where the snapshot is stored. dataDir=/tmp/zookeeper # the port at which the clients will connect clientPort=2181 secureClientPort=2182 authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.trustStore.location=kafka.zookeeper.truststore.jks ssl.trustStore.password=123456 ssl.keyStore.location=kafka.zookeeper.keystore.jks ssl.keyStore.password=123456 ssl.clientAuth=need # disable the per-ip limit on the number of connections since this is a non-production config maxClientCnxns=0 # Disable the adminserver by default to avoid port conflicts. # Set the port to something non-conflicting if choosing to enable this admin.enableServer=false # admin.serverPort=8080 #5 Test the SSL kafka@zoo:/etc/kafka$ openssl s_client -showcerts -connect localhost:2182 CONNECTED(00000003) Can't use SSL_get_servername depth=1 C = AU, ST = Melbourne, O = Internet Widgits Pty Ltd verify error:num=19:self signed certificate in certificate chain verify return:1 depth=1 C = AU, ST = Melbourne, O = Internet Widgits Pty Ltd verify return:1 depth=0 C = Unknown, ST = Unknown, L = Budapest, O = K Inc., OU = K, CN = John verify return:1 140307772192064:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42 --- Certificate chain 0 s:C = Unknown, ST = Unknown, L = Budapest, O = K Inc., OU = K, CN = John i:C = AU, ST = Melbourne, O = Internet Widgits Pty Ltd -----BEGIN CERTIFICATE----- MIIDLjCCAhYCFDmPPYOIc+LFjLje1r6oWDALW4w9MA0GCSqGSIb3DQEBCwUAMEQx CzAJBgNVBAYTAkFVMRIwEAYDVQQIDAlNZWxib3VybmUxITAfBgNVBAoMGEludGVy bmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMTA3MzAwMjQ4MTJaFw0zMTA3MjgwMjQ4 MTJaMGMxEDAOBgNVBAYTB1Vua25vd24xEDAOBgNVBAgTB1Vua25vd24xETAPBgNV BAcTCEJ1ZGFwZXN0MQ8wDQYDVQQKEwZLIEluYy4xCjAIBgNVBAsTAUsxDTALBgNV BAMTBEpvaG4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDXvPlp6/yp 3S2QQtumEdLFamTRHt6sZVXG1jT4fWMj5gWSaqFKtnjn0At334qycG7IQtY/9Qnf H3OCghpjN3P5h/ImGQKDbL9E4AhE2SI6kz5MRO7HRhpneRloUwuSqWQr0ARVaC12 e6EOGIhZlSMer68A/tOCqYKy95ZOWRugwSeghNyEeV1hi7buew9N1tpVKxa01CoB tFwwiKbcXAhX4Ea0pnOQxDF6QmrMPVzrt03PEkBxpm/t/Hl4fdxt4UuuldNcyiPN wdj/zjwFAqBxI9xMo5zzxh7bxT4uv8Ij701GXZr0jKfsV97KMiIJ5fNMlDf3awUa RfLFVbph4tuPAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAFc9IVeyqdIlrEjtT9kc vLrLCXy3mJuxmpxN7SnAOeNGqWkOonjIXmumQUiuBGk++F13a0+yD3JUN5o+zpVE k5ptk2nPsWkrhWaR6t1OG5ucdNHNtNE49tY1eqWBHOulnyVih2D52fXhUHB78Rtc vXbgXHOXDoLznCPe0JcHcuvtruiiKr0KTjABBlVQogxkUDq9ttvXqb1lUtosqPgG aSHOpEE0sRi+YfHQv6ludK982O/bfc8FZu0xJwp9PIMwFkXMoUyp7gxoFQtncTaM E8Mz8OekXT8sBQ5WBoArs9LRbeQfot2uhTA0/FfUK3v7r2twHwfRK9Zl6SiDV5Pk XkA= -----END CERTIFICATE----- 1 s:C = AU, ST = Melbourne, O = Internet Widgits Pty Ltd i:C = AU, ST = Melbourne, O = Internet Widgits Pty Ltd -----BEGIN CERTIFICATE----- MIIDaTCCAlGgAwIBAgIUfsM/BlsOkJyd5IrQBNOFq0lsx08wDQYJKoZIhvcNAQEL BQAwRDELMAkGA1UEBhMCQVUxEjAQBgNVBAgMCU1lbGJvdXJuZTEhMB8GA1UECgwY SW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMB4XDTIxMDczMDAyNDQ0NloXDTMxMDcy ODAyNDQ0NlowRDELMAkGA1UEBhMCQVUxEjAQBgNVBAgMCU1lbGJvdXJuZTEhMB8G A1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIIBIjANBgkqhkiG9w0BAQEF AAOCAQ8AMIIBCgKCAQEAsaUHS0ZXo4y4vBvPyhnI7MVyXppZmZW4laqniglQLZAZ fJQBApfDH9hQJQTNZQ6bPkukPOFGbJ3+LPBlmdojk6Rio9cV05FqAN2iH6Gbbn9a IPkSyBBbCsn8Y1Mq5scfMzgSAJfW8dw3GixDzf/+I1ZkPRkfk9A1K0tJLxmb4zcR J9LPIA90Qqr807CFgFbasq1d8e8hrKVkmpoC6gZCGxVbACZvvW3RJ9uOicDql99u UbPEGY8JjxN8zgubEkdIEXBFJxeic9HCgkfeydK1YJvVLAnIWAyCva8smoVGShu5 IvdNYFcKcbC+s1i4EdHUDauRJwMhSZ+ZmjZGl/3RpwIDAQABo1MwUTAdBgNVHQ4E FgQUldMJGkGoz78mBgJq9CAow0lTBaQwHwYDVR0jBBgwFoAUldMJGkGoz78mBgJq 9CAow0lTBaQwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAQn73 Zk+L78Gy9v7/xu2EaJjt+PEb0WhP6cknUs/MEwBJoTH1VYsUK4cqZ7PA5TDbRfM0 UHnmQuS5A9MP3eeqIUllp8YusBl4bYxIkS+5JJwmtLSlcE3JPtBmgSo1c+GKnnCh +9CfGqC7e2YNjwo1wYAgErAKV8RNh49KFn1/oAr8iT18J2pEvnQE5cU+uv7esOy7 DwwVEVOkqVfWAhlI3YcK8Q7cUMJpF0ksFVHJwn05mPI3gmAnK4M6FxtMg6T+IdUE gJp/njW8pf/ZnF9t4Up+qD/IHsLAqv9EHBgDaTJcpTG9mh/s9HeqGaYSv2f5pIxk eJyw+IlbMjZ4kLgoKw== -----END CERTIFICATE----- --- Server certificate subject=C = Unknown, ST = Unknown, L = Budapest, O = K Inc., OU = K, CN = John issuer=C = AU, ST = Melbourne, O = Internet Widgits Pty Ltd --- Acceptable client certificate CA names C = AU, ST = Melbourne, O = Internet Widgits Pty Ltd Client Certificate Types: ECDSA sign, RSA sign, DSA sign Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 2231 bytes and written 388 bytes Verification error: self signed certificate in certificate chain --- New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: F9EBF583C0FE33D2AEB8FF1CF81A1B79A577D0FFC8D6B028D4FE5020F5554EE1 Session-ID-ctx: Master-Key: 2A655AEB537244E0129B8967A6942488C999B7A02CCD43E7707D2DD8E0A366CC673D843E945A584A8834988BFD9B8141 PSK identity: None PSK identity hint: None SRP username: None Start Time: 1627615015 Timeout : 7200 (sec) Verify return code: 19 (self signed certificate in certificate chain) Extended master secret: yes --- # 6 Logs from zookeeper: [2021-07-30 11:17:32,076] ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [2021-07-30 11:17:32,090] WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1565) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1579) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1463) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1290) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1337) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ... 17 more Best regards, John Mark Causing
