Hi Team,
We have a 3 node kafka-zookeeper cluster setup with kafka-zookeeper
communicating on SSL.
We are currently using apache kafka 2.5 and zookeeper 3.5.7 . We are
trying to increase the replication factor in kafka topics using the
below method:
To increase the number of replicas for a given topic you have to:
1. Specify the extra replicas in a custom reassignment json file
For example, you could create increase-replication-factor.json and put
this content in it:
{"version":1,
"partitions":[
{"topic":"signals","partition":0,"replicas":[0,1,2]},
{"topic":"signals","partition":1,"replicas":[0,1,2]},
{"topic":"signals","partition":2,"replicas":[0,1,2]}
]}
2. Use the file with the --execute option of the kafka-reassign-partitions tool
[or kafka-reassign-partitions.sh - depending on the kafka package]
For example:
$ kafka-reassign-partitions --zookeeper localhost:2182
--reassignment-json-file increase-replication-factor.json --execute
--command-config zookeeper_client.properties
But we are facing the problem while running the
kafka-reassign-partitions , while running this command the connection
to zookeeper fails with below error:
2022-04-28 05:56:46,963 [myid:1] - ERROR
[nioEventLoopGroup-7-3:NettyServerCnxnFactory$CertificateVerifier@363]
- Unsuccessful handshake with session 0x0 2022-04-28 05:56:46,963
[myid:1] - WARN
[nioEventLoopGroup-7-3:NettyServerCnxnFactory$CnxnChannelHandler@220]
- Exception caught io.netty.handler.codec.DecoderException:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source) Caused by:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1198)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
We are passing all the certificate and keystore data through
--command-config , the zookeeper_client.properties is as below:
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2
zookeeper.ssl.truststore.location=kafka.truststore.jks
zookeeper.ssl.truststore.password=changeme
zookeeper.ssl.keystore.location=kafka.keystore.jks
zookeeper.ssl.keystore.password=changeme
zookeeper.ssl.endpoint.identification.algorithm=
zookeeper.ssl.hostnameVerification=false
We have also tried to set CLIENT_JVMFLAGS and KAFKA_OPTS with same jvm
arguments but that doesn't help.
The option of passing zookeeper_client.properties via
-zk-tls-config-file is not available in kafka-reassign-partitions.sh.
Can some please help how we can solve the issue.
Thanks
--
<https://www.pingidentity.com>[image: Ping Identity]
<https://www.pingidentity.com>
Ankit Bhalla
Senior Software Engineer
abha...@pingidentity.com
Connect with us: [image: Glassdoor logo]
<https://www.glassdoor.com/Overview/Working-at-Ping-Identity-EI_IE380907.11,24.htm>
[image:
LinkedIn logo] <https://www.linkedin.com/company/21870> [image: twitter
logo] <https://twitter.com/pingidentity> [image: facebook logo]
<https://www.facebook.com/pingidentitypage> [image: youtube logo]
<https://www.youtube.com/user/PingIdentityTV> [image: Blog logo]
<https://www.pingidentity.com/en/blog.html>
<https://www.gartner.com/reviews/vendor/write/ping-identity/?utm_content=vlp-write&refVal=vlp-ping-identity-32202&utm_campaign=vendor&utm_source=ping-identity&utm_medium=web&arwol=false>
--
_CONFIDENTIALITY NOTICE: This email may contain confidential and privileged
material for the sole use of the intended recipient(s). Any review, use,
distribution or disclosure by others is strictly prohibited. If you have
received this communication in error, please notify the sender immediately
by e-mail and delete the message and any file attachments from your
computer. Thank you._
