Hi Rajini Now that 0.11.0 is out, can we use the Admin client ? Are there some example code for these ?
Thanks. On Wed, May 24, 2017 at 9:06 PM, Rajini Sivaram <rajinisiva...@gmail.com> wrote: > Hi Raghav, > > Yes, you can create ACLs programmatically. Take a look at the use of > AclCommand.main in https://github.com/apache/kafka/blob/trunk/core/src/ > test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala > > If you can wait for the next release 0.11.0 that will be out next month, > you can use the new Java AdminClient, which allows you to do this in a much > neater way. Take a look at the interface https://github.com/ > apache/kafka/blob/trunk/clients/src/main/java/org/ > apache/kafka/clients/admin/AdminClient.java > <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java> > > If your release is not imminent, then you could build Kafka from the > 0.11.0 branch and use the new AdminClient. When the release is out, you can > switch over to the binary release. > > Regards, > > Rajini > > > > On Wed, May 24, 2017 at 4:13 PM, Raghav <raghavas...@gmail.com> wrote: > >> Hi Rajini >> >> Quick question on Configuring ACLs: We used bin/kafka-acls.sh to >> configure ACL rules, which internally uses Kafka Admin APIs to configure >> the ACLs. >> >> Can I add, remove and list ACLs via zk client libraries ? I want to be >> able to add, remove, list ACLs via my code rather than using Kafka-acl.sh. >> Is there a guideline for recommended set of libraries to use to do such >> operations ? >> >> As always thanks so much. >> >> >> >> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <rajinisiva...@gmail.com> >> wrote: >> >>> Raghav/Darshan, >>> >>> Can you try these steps on a clean installation of Kafka? It works for >>> me, so hopefully it will work for you. And then you can adapt to your >>> scenario. >>> >>> *Create keystores and truststores:* >>> >>> keytool -genkey -alias kafka -keystore server.keystore.jks -dname >>> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password >>> -keypass server-key-password >>> >>> keytool -exportcert -file server-cert-file -keystore server.keystore.jks >>> -alias kafka -storepass server-keystore-password >>> >>> keytool -importcert -file server-cert-file -keystore >>> server.truststore.jks -alias kafka -storepass server-truststore-password >>> -noprompt >>> >>> keytool -importcert -file server-cert-file -keystore >>> client.truststore.jks -alias kafkaclient -storepass >>> client-truststore-password -noprompt >>> >>> >>> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname >>> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password >>> -keypass client-key-password >>> >>> keytool -exportcert -file client-cert-file -keystore client.keystore.jks >>> -alias kafkaclient -storepass client-keystore-password >>> >>> keytool -importcert -file client-cert-file -keystore >>> server.truststore.jks -alias kafkaclient -storepass >>> server-truststore-password -noprompt >>> >>> *Configure broker: Add these lines at the end of your server.properties* >>> >>> listeners=SSL://:9093 >>> >>> advertised.listeners=SSL://127.0.0.1:9093 >>> >>> ssl.keystore.location=/tmp/acl/server.keystore.jks >>> >>> ssl.keystore.password=server-keystore-password >>> >>> ssl.key.password=server-key-password >>> >>> ssl.truststore.location=/tmp/acl/server.truststore.jks >>> >>> ssl.truststore.password=server-truststore-password >>> >>> security.inter.broker.protocol=SSL >>> >>> security.protocol=SSL >>> >>> ssl.client.auth=required >>> >>> allow.everyone.if.no.acl.found=false >>> >>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer >>> >>> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK >>> >>> *Configure producer: producer.properties* >>> >>> security.protocol=SSL >>> >>> ssl.truststore.location=/tmp/acl/client.truststore.jks >>> >>> ssl.truststore.password=client-truststore-password >>> >>> ssl.keystore.location=/tmp/acl/client.keystore.jks >>> >>> ssl.keystore.password=client-keystore-password >>> >>> ssl.key.password=client-key-password >>> >>> >>> *Configure consumer: consumer.properties* >>> >>> security.protocol=SSL >>> >>> ssl.truststore.location=/tmp/acl/client.truststore.jks >>> >>> ssl.truststore.password=client-truststore-password >>> >>> ssl.keystore.location=/tmp/acl/client.keystore.jks >>> >>> ssl.keystore.password=client-keystore-password >>> >>> ssl.key.password=client-key-password >>> >>> group.id=testgroup >>> >>> *Create topic:* >>> >>> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic >>> --replication-factor 1 --partitions 1 >>> >>> >>> *Configure ACLs:* >>> >>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 >>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer >>> --topic testtopic >>> >>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 >>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer >>> --topic testtopic --group test group >>> >>> >>> *Run console producer and type in some messages:* >>> >>> bin/kafka-console-producer.sh --producer.config >>> /tmp/acl/producer.properties --topic testtopic --broker-list >>> 127.0.0.1:9093 >>> >>> >>> *Run console consumer, you should see messages from above:* >>> >>> bin/kafka-console-consumer.sh --consumer.config >>> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server >>> 127.0.0.1:9093 --from-beginning >>> >>> >>> >>> On Tue, May 23, 2017 at 12:57 PM, Raghav <raghavas...@gmail.com> wrote: >>> >>>> Darshan, >>>> >>>> I have not yet successfully gotten the ACLs to work in Kafka. I am still >>>> looking for help. I will update this email thread if I do find. In case >>>> you >>>> get it working, please let me know. >>>> >>>> Thanks. >>>> >>>> R >>>> >>>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare < >>>> purandare.dars...@gmail.com> wrote: >>>> >>>> > Raghav >>>> > >>>> > I saw few posts of yours around Kafka ACLs and the problems. I have >>>> seen >>>> > similar issues where Writer has not been able to write to any topic. >>>> I have >>>> > seen "leader not available" and sometimes "unknown topic or >>>> partition", and >>>> > "topic_authorization_failed" error. >>>> > >>>> > Let me know if you find a valid config that works. >>>> > >>>> > Thanks. >>>> > >>>> > >>>> > >>>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <raghavas...@gmail.com> >>>> wrote: >>>> > >>>> >> Hello Kafka Users >>>> >> >>>> >> I am a new Kafka user and trying to make Kafka SSL work with >>>> Authorization >>>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to >>>> the >>>> >> point but my producer cannot write to kafka broker. I get >>>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors. >>>> >> >>>> >> Can someone please share their config which worked with ACLs. >>>> >> >>>> >> Here is my config. Please help. >>>> >> >>>> >> server.properties config >>>> >> ------------------------------------------------------------ >>>> >> ------------------------------------------------ >>>> >> broker.id=0 >>>> >> auto.create.topics.enable=true >>>> >> delete.topic.enable=true >>>> >> >>>> >> listeners=PLAINTEXT://kafka1.example.com:9092 >>>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093 >>>> >> <http://kafka-dev1.example.com:9093/> >>>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/> >>>> >> >>>> >> >>>> >> >>>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks >>>> >> ssl.keystore.password=12345678 >>>> >> ssl.key.password=12345678 >>>> >> >>>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks >>>> >> ssl.truststore.password=12345678 >>>> >> >>>> >> ssl.client.auth=required >>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 >>>> >> ssl.keystore.type=JKS >>>> >> ssl.truststore.type=JKS >>>> >> >>>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer >>>> >> ------------------------------------------------------------ >>>> >> ------------------------------------------------ >>>> >> >>>> >> >>>> >> >>>> >> Here is producer Config(producer.properties) >>>> >> ------------------------------------------------------------ >>>> >> ------------------------------------------------ >>>> >> security.protocol=SSL >>>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks >>>> >> ssl.truststore.password=12345678 >>>> >> >>>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks >>>> >> ssl.keystore.password=12345678 >>>> >> ssl.key.password=12345678 >>>> >> >>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 >>>> >> ssl.truststore.type=JKS >>>> >> ssl.keystore.type=JKS >>>> >> >>>> >> ------------------------------------------------------------ >>>> >> ------------------------------------------------ >>>> >> >>>> >> >>>> >> Raqhav >>>> >> >>>> > >>>> > >>>> >>>> >>>> -- >>>> Raghav >>>> >>> >>> >> >> >> -- >> Raghav >> > > -- Raghav
