Hi Alex Brekken,
Sorry for the delayed response, I tried your fix,
At first I got
> Fatal error during KafkaServer startup. Prepare to shutdown"
> "org.apache.kafka.common.KafkaException: Exception while determining if
> ZooKeeper is secure
> [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
> zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
> at
> org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:75)
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:441)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> at kafka.Kafka$.main(Kafka.scala:109)
> at kafka.Kafka.main(Kafka.scala)
Then I set zookeeper.sasl.client=true
> - 10.91.21.142 arjun-8481 - - - 23
> org.apache.zookeeper.client.ZooKeeperSaslClient respondToServer SEVERE
> "23-11-2023 10:49:21:770" - "SASL authentication failed using login context
> 'Client' with exception: {}" "javax.security.sasl.SaslException: Error in
> authenticating with a Zookeeper Quorum member: the quorum member's
> saslToken is null.
> at
> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
> at
> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
> - 10.91.21.142 arjun-8481 - - - 23
> org.apache.zookeeper.ClientCnxn$SendThread run INFO "23-11-2023
> 10:49:21:771" - "Unable to read additional data from server sessionid
> 0x100147c3ccb0000, likely server has closed socket, closing socket
> connection and attempting reconnect" - - - - - - 1700716761771 - - - - - -
> - - logger_name=org.apache.zookeeper.ClientCnxn
> - 10.91.21.142 arjun-8481 - - - 24 kafka.utils.Logging error SEVERE
> "23-11-2023 10:49:21:771" - "[ZooKeeperClient Kafka server] Auth failed." -
> - - - - - 1700716761771 - - - - - - - - logtype=application
> logger_name=kafka.zookeeper.ZooKeeperClient
> - 10.91.21.142 arjun-8481 - - - 24
> org.apache.zookeeper.ClientCnxn$EventThread run INFO "23-11-2023
> 10:49:21:773" - "EventThread shut down for session: 0x100147c3ccb0000" - -
> - - - - 1700716761773 - - - - - - - - logtype=application
> thread_name=main-EventThread logger_name=org.apache.zookeeper.ClientCnxn
> - 10.91.21.142 arjun-8481 - - - 1 kafka.utils.Logging fatal SEVERE
> "23-11-2023 10:49:21:887" - "Fatal error during KafkaServer startup.
> Prepare to shutdown"
> "org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode
> = AuthFailed for /kafka/1/kafka/1
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at
> kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:583)
> at
> kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
> at
> kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627)
> at
> kafka.server.KafkaServer.$anonfun$initZkClient$2(KafkaServer.scala:451)
> at
> kafka.server.KafkaServer.$anonfun$initZkClient$2$adapted(KafkaServer.scala:448)
> at scala.Option.foreach(Option.scala:437)
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:448)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> at kafka.Kafka$.main(Kafka.scala:109)
> at kafka.Kafka.main(Kafka.scala)
Please advice.
On Mon, Nov 13, 2023 at 4:11 AM Alex Brekken <brek...@gmail.com> wrote:
> I see a couple of things that look wrong. First, remove this line from your
> ZK config:
>
> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider.
> And replace it with this:
>
> authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider.
>
> Additionally, I think you need to add these lines to the ZK config if you
> want ZK to ZK authentication:
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=true
> quorum.auth.serverRequireSasl=true
>
> The rest looks OK.
>
> On Sat, Nov 11, 2023 at 9:12 PM arjun s v <arjun.cs...@gmail.com> wrote:
>
> > A small correction, I'm not trying to enable mTLS, just a simple
> > authentication(Digest or Plain) is enough,
> > Sharing the jaas files and config files,
> > kafka_server_jaas.conf
> >
> > > admin.KafkaServer{
> > >
> > > org.apache.kafka.common.security.plain.PlainLoginModule required
> > >
> > > username="USERNAME"
> > >
> > > password="PASSWORD";
> > >
> > > };
> > >
> > > KafkaServer{
> > >
> > > org.apache.kafka.common.security.plain.PlainLoginModule required
> > >
> > > username="USERNAME"
> > >
> > > password="PASSWORD";
> > >
> > > };
> > >
> > > Client{
> > >
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > >
> > > username="super"
> > >
> > > password="adminsecret";
> > >
> > > };
> > >
> > server properties
> >
> > > zookeeper.sasl.client=false
> >
> >
> java.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf
> >
> > zookeeper.connection.timeout.ms=6000
> >
> > zookeeper.sync.time.ms=2000
> >
> > zookeeper.set.acl=true
> >
> >
> > zk_server_jaas.conf
> >
> > > QuorumServer{
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > > user_test="test";
> > > };
> > > QuorumLearner{
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > > username="test"
> > > password="test";
> > > };
> > > Server{
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > > username="super"
> > > password="adminsecret";
> >
> > };
> >
> >
> > zoo.cfg
> >
> > >
> > >
> >
> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider
> >
> >
> > env
> >
> > > SERVER_JVMFLAGS="$SERVER_JVMFLAGS
> > > -Djava.security.auth.login.config=$ZOOCFGDIR/zk_server_jaas.conf"
> > >
> > On Fri, Nov 10, 2023 at 7:03 PM Alex Brekken <brek...@gmail.com> wrote:
> >
> > > Ok, so you're trying to enable both SASL authentication (digest) and
> TLS,
> > > using mTLS for Zookeeper? I'm just trying to understand the bigger
> > > picture. The error you're getting regarding the Sasl token sounds like
> > > either the jaas config on the Kafka broker side is wrong/missing, or
> the
> > > jaas config on the ZK side is wrong/missing. (you need both - in this
> > case
> > > the broker is the "client" and ZK is the "server"). Are you able to
> share
> > > the jaas config you're using for both Kafka and ZK? Without seeing
> that
> > > it's tough to know. Also, to make troubleshooting easier you might
> want
> > to
> > > leave TLS out of it for now and get SASL working first. (or
> vice-versa)
> > >
> > >
> > >
> > > On Thu, Nov 9, 2023 at 11:26 PM arjun s v <arjun.cs...@gmail.com>
> wrote:
> > >
> > > > "Digest-MD5 is SASL authentication, so not sure what you mean here."
> > > > If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken"
> > and
> > > > throws the following error,
> > > >
> > > > "SASL authentication failed using login context 'Client' with
> > exception:
> > > > {}" "javax.security.sasl.SaslException: Error in authenticating with
> a
> > > > Zookeeper Quorum member: the quorum member's saslToken is null.
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
> > > >
> > > > at
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
> > > >
> > > >
> > > > "Hmm, that config shouldn't have anything to do with TLS. You can set
> > > ACL's
> > > >
> > > > with or without TLS encryption. Were you getting an error?"
> > > >
> > > >
> > > > "Fatal error during KafkaServer startup. Prepare to shutdown"
> > > > "java.lang.SecurityException: zookeeper.set.acl is true, but
> ZooKeeper
> > > > client TLS configuration identifying at least
> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
> > > > kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
> > > > present and the verification of the JAAS login file failed
> > > > [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
> > > > zookeeper.sasl.client=false,
> > zookeeper.sasl.clientconfig=default:Client]
> > > >
> > > > at
> kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
> > > >
> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> > > >
> > > > at kafka.Kafka$.main(Kafka.scala:109)
> > > >
> > > > at kafka.Kafka.main(Kafka.scala)
> > > >
> > > >
> > > > "This was the 2nd result in a google search:
> > > > https://docs.confluent.io/platform/current/security/zk-security.html
> "
> > > >
> > > > FYKI, I've googled, asked chat gpt, surfed over many zookeeper and
> > kafka
> > > > docs and blog,
> > > > I remember trying the doc you suggested here about 10 days back in
> the
> > > > initial days of this task!
> > > > About the doc you suggested,
> > > > I cannot configure SSL as I already mentioned, If I skip ssl config
> > part
> > > > from your suggested doc and tried Digest-MD5, I come up "saslToken
> > > missing"
> > > > exception which I mentioned above!
> > > > I don't really understand what saslToken is and how to make it get
> > > > generated for Digest auth!
> > > > Please assist!
> > > >
> > > > On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <alexcrai...@gmail.com>
> > wrote:
> > > >
> > > > > " I couldn't find any doc by kafka to enable Digest-MD5
> > > authentication."
> > > > > This was the 2nd result in a google search:
> > > > >
> https://docs.confluent.io/platform/current/security/zk-security.html
> > > > >
> > > > > " I don't want to enable SASL."
> > > > > Digest-MD5 is SASL authentication, so not sure what you mean here.
> > > > >
> > > > > " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
> > > > > Hmm, that config shouldn't have anything to do with TLS. You can
> set
> > > > ACL's
> > > > > with or without TLS encryption. Were you getting an error?
> > > > >
> > > > > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <arjun.cs...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Team,
> > > > > >
> > > > > > Please consider this as high priority, we need to enable
> > > authentication
> > > > > > ASAP. Please assist.
> > > > > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <arjun.cs...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > I'm trying to configure *Digest-MD5* authentication between
> kafka
> > > and
> > > > > > > zookeeper.
> > > > > > > Also I need to set ACL with digest scheme and credentials.
> > > > > > > I don't want to enable SASL.
> > > > > > > I tried to follow this
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> > > > > >
> > > > > > doc
> > > > > > > from zookeeper,
> > > > > > >
> > > > > > > - If I configured a jaas file, I have to set
> > > > > > zookeeper.sasl.client=true(if
> > > > > > > not kafka throws error from JaasUtils) which enables sasl
> > > > > > authentication.
> > > > > > > - If I set zookeeper.set.acl=true, I'm forced to configure
> > TLS.
> > > > > > >
> > > > > > > I couldn't find any doc by kafka to enable Digest-MD5
> > > authentication.
> > > > > > > I cannot configure kerberos or TLS, just a Digest-MD5 is
> > sufficient
> > > > for
> > > > > > my
> > > > > > > usecase.
> > > > > > > Please let me know if there are any docs to enable Digest-MD5
> > auth
> > > > > > between
> > > > > > > kafka and zookeeper.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Arjun S V
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
