Hi Alex Brekken, Sorry for the delayed response, I tried your fix, At first I got
> Fatal error during KafkaServer startup. Prepare to shutdown" > "org.apache.kafka.common.KafkaException: Exception while determining if > ZooKeeper is secure > [java.security.auth.login.config=./../config/kafka_server_jaas.conf, > zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client] > at > org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:75) > at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:441) > at kafka.server.KafkaServer.startup(KafkaServer.scala:191) > at kafka.Kafka$.main(Kafka.scala:109) > at kafka.Kafka.main(Kafka.scala) Then I set zookeeper.sasl.client=true > - 10.91.21.142 arjun-8481 - - - 23 > org.apache.zookeeper.client.ZooKeeperSaslClient respondToServer SEVERE > "23-11-2023 10:49:21:770" - "SASL authentication failed using login context > 'Client' with exception: {}" "javax.security.sasl.SaslException: Error in > authenticating with a Zookeeper Quorum member: the quorum member's > saslToken is null. > at > org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312) > at > org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275) > at > org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) > - 10.91.21.142 arjun-8481 - - - 23 > org.apache.zookeeper.ClientCnxn$SendThread run INFO "23-11-2023 > 10:49:21:771" - "Unable to read additional data from server sessionid > 0x100147c3ccb0000, likely server has closed socket, closing socket > connection and attempting reconnect" - - - - - - 1700716761771 - - - - - - > - - logger_name=org.apache.zookeeper.ClientCnxn > - 10.91.21.142 arjun-8481 - - - 24 kafka.utils.Logging error SEVERE > "23-11-2023 10:49:21:771" - "[ZooKeeperClient Kafka server] Auth failed." - > - - - - - 1700716761771 - - - - - - - - logtype=application > logger_name=kafka.zookeeper.ZooKeeperClient > - 10.91.21.142 arjun-8481 - - - 24 > org.apache.zookeeper.ClientCnxn$EventThread run INFO "23-11-2023 > 10:49:21:773" - "EventThread shut down for session: 0x100147c3ccb0000" - - > - - - - 1700716761773 - - - - - - - - logtype=application > thread_name=main-EventThread logger_name=org.apache.zookeeper.ClientCnxn > - 10.91.21.142 arjun-8481 - - - 1 kafka.utils.Logging fatal SEVERE > "23-11-2023 10:49:21:887" - "Fatal error during KafkaServer startup. > Prepare to shutdown" > "org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode > = AuthFailed for /kafka/1/kafka/1 > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:130) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:54) > at > kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:583) > at > kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729) > at > kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627) > at > kafka.server.KafkaServer.$anonfun$initZkClient$2(KafkaServer.scala:451) > at > kafka.server.KafkaServer.$anonfun$initZkClient$2$adapted(KafkaServer.scala:448) > at scala.Option.foreach(Option.scala:437) > at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:448) > at kafka.server.KafkaServer.startup(KafkaServer.scala:191) > at kafka.Kafka$.main(Kafka.scala:109) > at kafka.Kafka.main(Kafka.scala) Please advice. On Mon, Nov 13, 2023 at 4:11 AM Alex Brekken <brek...@gmail.com> wrote: > I see a couple of things that look wrong. First, remove this line from your > ZK config: > > zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider. > And replace it with this: > > authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider. > > Additionally, I think you need to add these lines to the ZK config if you > want ZK to ZK authentication: > quorum.auth.enableSasl=true > quorum.auth.learnerRequireSasl=true > quorum.auth.serverRequireSasl=true > > The rest looks OK. > > On Sat, Nov 11, 2023 at 9:12 PM arjun s v <arjun.cs...@gmail.com> wrote: > > > A small correction, I'm not trying to enable mTLS, just a simple > > authentication(Digest or Plain) is enough, > > Sharing the jaas files and config files, > > kafka_server_jaas.conf > > > > > admin.KafkaServer{ > > > > > > org.apache.kafka.common.security.plain.PlainLoginModule required > > > > > > username="USERNAME" > > > > > > password="PASSWORD"; > > > > > > }; > > > > > > KafkaServer{ > > > > > > org.apache.kafka.common.security.plain.PlainLoginModule required > > > > > > username="USERNAME" > > > > > > password="PASSWORD"; > > > > > > }; > > > > > > Client{ > > > > > > org.apache.zookeeper.server.auth.DigestLoginModule required > > > > > > username="super" > > > > > > password="adminsecret"; > > > > > > }; > > > > > server properties > > > > > zookeeper.sasl.client=false > > > > > java.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf > > > > zookeeper.connection.timeout.ms=6000 > > > > zookeeper.sync.time.ms=2000 > > > > zookeeper.set.acl=true > > > > > > zk_server_jaas.conf > > > > > QuorumServer{ > > > org.apache.zookeeper.server.auth.DigestLoginModule required > > > user_test="test"; > > > }; > > > QuorumLearner{ > > > org.apache.zookeeper.server.auth.DigestLoginModule required > > > username="test" > > > password="test"; > > > }; > > > Server{ > > > org.apache.zookeeper.server.auth.DigestLoginModule required > > > username="super" > > > password="adminsecret"; > > > > }; > > > > > > zoo.cfg > > > > > > > > > > > zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider > > > > > > env > > > > > SERVER_JVMFLAGS="$SERVER_JVMFLAGS > > > -Djava.security.auth.login.config=$ZOOCFGDIR/zk_server_jaas.conf" > > > > > On Fri, Nov 10, 2023 at 7:03 PM Alex Brekken <brek...@gmail.com> wrote: > > > > > Ok, so you're trying to enable both SASL authentication (digest) and > TLS, > > > using mTLS for Zookeeper? I'm just trying to understand the bigger > > > picture. The error you're getting regarding the Sasl token sounds like > > > either the jaas config on the Kafka broker side is wrong/missing, or > the > > > jaas config on the ZK side is wrong/missing. (you need both - in this > > case > > > the broker is the "client" and ZK is the "server"). Are you able to > share > > > the jaas config you're using for both Kafka and ZK? Without seeing > that > > > it's tough to know. Also, to make troubleshooting easier you might > want > > to > > > leave TLS out of it for now and get SASL working first. (or > vice-versa) > > > > > > > > > > > > On Thu, Nov 9, 2023 at 11:26 PM arjun s v <arjun.cs...@gmail.com> > wrote: > > > > > > > "Digest-MD5 is SASL authentication, so not sure what you mean here." > > > > If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken" > > and > > > > throws the following error, > > > > > > > > "SASL authentication failed using login context 'Client' with > > exception: > > > > {}" "javax.security.sasl.SaslException: Error in authenticating with > a > > > > Zookeeper Quorum member: the quorum member's saslToken is null. > > > > > > > > at > > > > > > > > > > > > > > org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312) > > > > > > > > at > > > > > > > > > > > > > > org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275) > > > > > > > > at > > > > > > > > > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882) > > > > > > > > at > > > > > > > > > > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103) > > > > > > > > at > > > > > > > > > > > > > > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365) > > > > > > > > at > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) > > > > > > > > > > > > "Hmm, that config shouldn't have anything to do with TLS. You can set > > > ACL's > > > > > > > > with or without TLS encryption. Were you getting an error?" > > > > > > > > > > > > "Fatal error during KafkaServer startup. Prepare to shutdown" > > > > "java.lang.SecurityException: zookeeper.set.acl is true, but > ZooKeeper > > > > client TLS configuration identifying at least > > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp, > > > > kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and > > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not > > > > present and the verification of the JAAS login file failed > > > > [java.security.auth.login.config=./../config/kafka_server_jaas.conf, > > > > zookeeper.sasl.client=false, > > zookeeper.sasl.clientconfig=default:Client] > > > > > > > > at > kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445) > > > > > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:191) > > > > > > > > at kafka.Kafka$.main(Kafka.scala:109) > > > > > > > > at kafka.Kafka.main(Kafka.scala) > > > > > > > > > > > > "This was the 2nd result in a google search: > > > > https://docs.confluent.io/platform/current/security/zk-security.html > " > > > > > > > > FYKI, I've googled, asked chat gpt, surfed over many zookeeper and > > kafka > > > > docs and blog, > > > > I remember trying the doc you suggested here about 10 days back in > the > > > > initial days of this task! > > > > About the doc you suggested, > > > > I cannot configure SSL as I already mentioned, If I skip ssl config > > part > > > > from your suggested doc and tried Digest-MD5, I come up "saslToken > > > missing" > > > > exception which I mentioned above! > > > > I don't really understand what saslToken is and how to make it get > > > > generated for Digest auth! > > > > Please assist! > > > > > > > > On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <alexcrai...@gmail.com> > > wrote: > > > > > > > > > " I couldn't find any doc by kafka to enable Digest-MD5 > > > authentication." > > > > > This was the 2nd result in a google search: > > > > > > https://docs.confluent.io/platform/current/security/zk-security.html > > > > > > > > > > " I don't want to enable SASL." > > > > > Digest-MD5 is SASL authentication, so not sure what you mean here. > > > > > > > > > > " If I set zookeeper.set.acl=true, I'm forced to configure TLS." > > > > > Hmm, that config shouldn't have anything to do with TLS. You can > set > > > > ACL's > > > > > with or without TLS encryption. Were you getting an error? > > > > > > > > > > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <arjun.cs...@gmail.com> > > > wrote: > > > > > > > > > > > Team, > > > > > > > > > > > > Please consider this as high priority, we need to enable > > > authentication > > > > > > ASAP. Please assist. > > > > > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <arjun.cs...@gmail.com> > > > > wrote: > > > > > > > > > > > > > Hi team, > > > > > > > > > > > > > > I'm trying to configure *Digest-MD5* authentication between > kafka > > > and > > > > > > > zookeeper. > > > > > > > Also I need to set ACL with digest scheme and credentials. > > > > > > > I don't want to enable SASL. > > > > > > > I tried to follow this > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication > > > > > > > > > > > > doc > > > > > > > from zookeeper, > > > > > > > > > > > > > > - If I configured a jaas file, I have to set > > > > > > zookeeper.sasl.client=true(if > > > > > > > not kafka throws error from JaasUtils) which enables sasl > > > > > > authentication. > > > > > > > - If I set zookeeper.set.acl=true, I'm forced to configure > > TLS. > > > > > > > > > > > > > > I couldn't find any doc by kafka to enable Digest-MD5 > > > authentication. > > > > > > > I cannot configure kerberos or TLS, just a Digest-MD5 is > > sufficient > > > > for > > > > > > my > > > > > > > usecase. > > > > > > > Please let me know if there are any docs to enable Digest-MD5 > > auth > > > > > > between > > > > > > > kafka and zookeeper. > > > > > > > > > > > > > > Regards, > > > > > > > Arjun S V > > > > > > > > > > > > > > > > > > > > > > > > > > > >