Hi Raghav, Yes, you should be able to use AdminClient from 0.11.0. Take a look at the Javadocs ( https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/admin/package-summary.html). The integration tests may be useful too ( https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/AdminClientIntegrationTest.scala , https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/SaslSslAdminClientIntegrationTest.scala ).
Regards, Rajini On Wed, Jul 5, 2017 at 4:10 PM, Raghav <raghavas...@gmail.com> wrote: > Hi Rajini > > Now that 0.11.0 is out, can we use the Admin client ? Are there some > example code for these ? > > Thanks. > > On Wed, May 24, 2017 at 9:06 PM, Rajini Sivaram <rajinisiva...@gmail.com> > wrote: > >> Hi Raghav, >> >> Yes, you can create ACLs programmatically. Take a look at the use of >> AclCommand.main in https://github.com/apache/kafk >> a/blob/trunk/core/src/test/scala/integration/kafka/api/ >> EndToEndAuthorizationTest.scala >> >> If you can wait for the next release 0.11.0 that will be out next month, >> you can use the new Java AdminClient, which allows you to do this in a much >> neater way. Take a look at the interface https://github.com/a >> pache/kafka/blob/trunk/clients/src/main/java/org/apache/ >> kafka/clients/admin/AdminClient.java >> <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java> >> >> If your release is not imminent, then you could build Kafka from the >> 0.11.0 branch and use the new AdminClient. When the release is out, you can >> switch over to the binary release. >> >> Regards, >> >> Rajini >> >> >> >> On Wed, May 24, 2017 at 4:13 PM, Raghav <raghavas...@gmail.com> wrote: >> >>> Hi Rajini >>> >>> Quick question on Configuring ACLs: We used bin/kafka-acls.sh to >>> configure ACL rules, which internally uses Kafka Admin APIs to configure >>> the ACLs. >>> >>> Can I add, remove and list ACLs via zk client libraries ? I want to be >>> able to add, remove, list ACLs via my code rather than using Kafka-acl.sh. >>> Is there a guideline for recommended set of libraries to use to do such >>> operations ? >>> >>> As always thanks so much. >>> >>> >>> >>> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <rajinisiva...@gmail.com >>> > wrote: >>> >>>> Raghav/Darshan, >>>> >>>> Can you try these steps on a clean installation of Kafka? It works for >>>> me, so hopefully it will work for you. And then you can adapt to your >>>> scenario. >>>> >>>> *Create keystores and truststores:* >>>> >>>> keytool -genkey -alias kafka -keystore server.keystore.jks -dname >>>> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password >>>> -keypass server-key-password >>>> >>>> keytool -exportcert -file server-cert-file -keystore >>>> server.keystore.jks -alias kafka -storepass server-keystore-password >>>> >>>> keytool -importcert -file server-cert-file -keystore >>>> server.truststore.jks -alias kafka -storepass server-truststore-password >>>> -noprompt >>>> >>>> keytool -importcert -file server-cert-file -keystore >>>> client.truststore.jks -alias kafkaclient -storepass >>>> client-truststore-password -noprompt >>>> >>>> >>>> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname >>>> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password >>>> -keypass client-key-password >>>> >>>> keytool -exportcert -file client-cert-file -keystore >>>> client.keystore.jks -alias kafkaclient -storepass client-keystore-password >>>> >>>> keytool -importcert -file client-cert-file -keystore >>>> server.truststore.jks -alias kafkaclient -storepass >>>> server-truststore-password -noprompt >>>> >>>> *Configure broker: Add these lines at the end of your server.properties* >>>> >>>> listeners=SSL://:9093 >>>> >>>> advertised.listeners=SSL://127.0.0.1:9093 >>>> >>>> ssl.keystore.location=/tmp/acl/server.keystore.jks >>>> >>>> ssl.keystore.password=server-keystore-password >>>> >>>> ssl.key.password=server-key-password >>>> >>>> ssl.truststore.location=/tmp/acl/server.truststore.jks >>>> >>>> ssl.truststore.password=server-truststore-password >>>> >>>> security.inter.broker.protocol=SSL >>>> >>>> security.protocol=SSL >>>> >>>> ssl.client.auth=required >>>> >>>> allow.everyone.if.no.acl.found=false >>>> >>>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer >>>> >>>> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK >>>> >>>> *Configure producer: producer.properties* >>>> >>>> security.protocol=SSL >>>> >>>> ssl.truststore.location=/tmp/acl/client.truststore.jks >>>> >>>> ssl.truststore.password=client-truststore-password >>>> >>>> ssl.keystore.location=/tmp/acl/client.keystore.jks >>>> >>>> ssl.keystore.password=client-keystore-password >>>> >>>> ssl.key.password=client-key-password >>>> >>>> >>>> *Configure consumer: consumer.properties* >>>> >>>> security.protocol=SSL >>>> >>>> ssl.truststore.location=/tmp/acl/client.truststore.jks >>>> >>>> ssl.truststore.password=client-truststore-password >>>> >>>> ssl.keystore.location=/tmp/acl/client.keystore.jks >>>> >>>> ssl.keystore.password=client-keystore-password >>>> >>>> ssl.key.password=client-key-password >>>> >>>> group.id=testgroup >>>> >>>> *Create topic:* >>>> >>>> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic >>>> --replication-factor 1 --partitions 1 >>>> >>>> >>>> *Configure ACLs:* >>>> >>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 >>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" >>>> --producer --topic testtopic >>>> >>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 >>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" >>>> --consumer --topic testtopic --group test group >>>> >>>> >>>> *Run console producer and type in some messages:* >>>> >>>> bin/kafka-console-producer.sh --producer.config >>>> /tmp/acl/producer.properties --topic testtopic --broker-list >>>> 127.0.0.1:9093 >>>> >>>> >>>> *Run console consumer, you should see messages from above:* >>>> >>>> bin/kafka-console-consumer.sh --consumer.config >>>> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server >>>> 127.0.0.1:9093 --from-beginning >>>> >>>> >>>> >>>> On Tue, May 23, 2017 at 12:57 PM, Raghav <raghavas...@gmail.com> wrote: >>>> >>>>> Darshan, >>>>> >>>>> I have not yet successfully gotten the ACLs to work in Kafka. I am >>>>> still >>>>> looking for help. I will update this email thread if I do find. In >>>>> case you >>>>> get it working, please let me know. >>>>> >>>>> Thanks. >>>>> >>>>> R >>>>> >>>>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare < >>>>> purandare.dars...@gmail.com> wrote: >>>>> >>>>> > Raghav >>>>> > >>>>> > I saw few posts of yours around Kafka ACLs and the problems. I have >>>>> seen >>>>> > similar issues where Writer has not been able to write to any topic. >>>>> I have >>>>> > seen "leader not available" and sometimes "unknown topic or >>>>> partition", and >>>>> > "topic_authorization_failed" error. >>>>> > >>>>> > Let me know if you find a valid config that works. >>>>> > >>>>> > Thanks. >>>>> > >>>>> > >>>>> > >>>>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <raghavas...@gmail.com> >>>>> wrote: >>>>> > >>>>> >> Hello Kafka Users >>>>> >> >>>>> >> I am a new Kafka user and trying to make Kafka SSL work with >>>>> Authorization >>>>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to >>>>> the >>>>> >> point but my producer cannot write to kafka broker. I get >>>>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors. >>>>> >> >>>>> >> Can someone please share their config which worked with ACLs. >>>>> >> >>>>> >> Here is my config. Please help. >>>>> >> >>>>> >> server.properties config >>>>> >> ------------------------------------------------------------ >>>>> >> ------------------------------------------------ >>>>> >> broker.id=0 >>>>> >> auto.create.topics.enable=true >>>>> >> delete.topic.enable=true >>>>> >> >>>>> >> listeners=PLAINTEXT://kafka1.example.com:9092 >>>>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093 >>>>> >> <http://kafka-dev1.example.com:9093/> >>>>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/> >>>>> >> >>>>> >> >>>>> >> >>>>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks >>>>> >> ssl.keystore.password=12345678 >>>>> >> ssl.key.password=12345678 >>>>> >> >>>>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks >>>>> >> ssl.truststore.password=12345678 >>>>> >> >>>>> >> ssl.client.auth=required >>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 >>>>> >> ssl.keystore.type=JKS >>>>> >> ssl.truststore.type=JKS >>>>> >> >>>>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer >>>>> >> ------------------------------------------------------------ >>>>> >> ------------------------------------------------ >>>>> >> >>>>> >> >>>>> >> >>>>> >> Here is producer Config(producer.properties) >>>>> >> ------------------------------------------------------------ >>>>> >> ------------------------------------------------ >>>>> >> security.protocol=SSL >>>>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks >>>>> >> ssl.truststore.password=12345678 >>>>> >> >>>>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks >>>>> >> ssl.keystore.password=12345678 >>>>> >> ssl.key.password=12345678 >>>>> >> >>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 >>>>> >> ssl.truststore.type=JKS >>>>> >> ssl.keystore.type=JKS >>>>> >> >>>>> >> ------------------------------------------------------------ >>>>> >> ------------------------------------------------ >>>>> >> >>>>> >> >>>>> >> Raqhav >>>>> >> >>>>> > >>>>> > >>>>> >>>>> >>>>> -- >>>>> Raghav >>>>> >>>> >>>> >>> >>> >>> -- >>> Raghav >>> >> >> > > > -- > Raghav >
