Transfer data between kerberized kafka clusters (different principals)

Hi, I am trying to transfer data between two kerberized kafka clusters. The brokers are running under different users on both clusters and so, I am providing a different Kerberos.service.name to consumer and producer. For security reasons and access management rules in places, having the same p

Re: Transfer data between kerberized kafka clusters (different principals)

Hi, >From Kafka 0.10.2.0, we can configure producer/consumer jaas configuration using "sasl.jaas.config" config property. Using this we can configure different principals. On Tue, Apr 24, 2018 at 10:58 PM, Zieger, Antoine < antoine.zie...@morganstanley.com> wrote: > Hi, > > I am trying to tran

RE: Transfer data between kerberized kafka clusters (different principals)

Hi, Thank you very much for this quick answer. I am not sure I understand where/how to use this property. I see an example here https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients but I don't understand how to use it while creating a Producer

Re: Transfer data between kerberized kafka clusters (different principals)

Yes. Sample example/format here: https://kafka.apache.org/documentation/#security_kerberos_sasl_clientconfig On Tue, Apr 24, 2018 at 11:30 PM, Zieger, Antoine < antoine.zie...@morganstanley.com> wrote: > Hi, > > Thank you very much for this quick answer. I am not sure I understand > where/how to

RE: Transfer data between kerberized kafka clusters (different principals)

Hi, Thanks for the link, I am sorry this might be a lack of java skills on my side but I still don’t understand how I can use it in a java class. The example is provided in case of a property file from what I understand. Would you mind providing a java example ? producerConfig.put("sasl.jaas.c

Re: Broker cannot start switch to Java9 - weird file system issue ?

A JIRA ticket would be appreciated. :) Ismael On Sat, Apr 21, 2018 at 12:51 AM, Enrico Olivelli wrote: > Il sab 21 apr 2018, 06:29 Ismael Juma ha scritto: > > > Hi Enrico, > > > > It is a real problem because it causes indexes to take a lot more disk > > space upfront. The sparsity is an impor

ConsumerCoordinator Offset commit failed on partition xxx at offset nnn

Hi, (I did not get any response from the Storm user group) We use a custom Kafka spout in our Apache Storm topology. When a machine in the Kafka cluster is bounced, we see the following error message in the logs: ERROR ConsumerCoordinator [thread-iii] - [Consumer clientId=consumer-1, grou

Re: Transfer data between kerberized kafka clusters (different principals)

like this: producerConfig.put("sasl.jaas.config", "com.sun.security.auth.module.Krb5LoginModule required " + "useTicketCache=false " + "renewTicket=true " + "serviceName=\"kafka\" " + "useKeyTab=true " + "keyTab=\"/home/test.keytab\" "