Many "Shrinking ISR for partition" all at once on a single broker

Hello Kafka Users, We experienced an issue recently where a single Kafka broker reported that, for all of the topic-partitions for which it was leader: [2017-05-16 22:48:47,760] INFO Partition [,] on broker 561: Shrinking ISR for partition [,] from ,,561 to 561 (kafka.cluster.Partition) This mes

Re: Order of punctuate() and process() in a stream processor

Thanks Sini! I intended to create a new JIRA, but than changed my mind and just picky-backed it to the existing one, as it's highly related and we might be able to tackle it in one effort. -Matthias On 5/18/17 12:12 AM, Peter Sinoros Szabo wrote: > Hi Michal, > > yes, I know its beyond the sco

Re: Securing Kafka - Keystore and Truststore question

Raghav, If you send me the full command sequence, I can take a look. Also, which JRE are you using? Regards, Rajini On Thu, May 18, 2017 at 12:19 PM, Raghav wrote: > Rajini > > I just tried this. It turns out that I can't import cert-file by itself in > trust store until it is signed by a CA.

Re: Securing Kafka - Keystore and Truststore question

Rajini I just tried this. It turns out that I can't import cert-file by itself in trust store until it is signed by a CA. Could be because of the format ? Any idea here ... In the above steps, if I sign the server-cert-file and client-cert-file by a private CA then I can add them to trust store a

Re: Securing Kafka - Keystore and Truststore question

Raghav, Perhaps what you want to do is: *You do (for the brokers):* Generate key-pair for broker: keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey Export certificate to a file to send to your customers: keytool -exportcert -file server-cert-file -keystore kaf

Topic creation fails on Kerberos/SASL enabled brokers

I have setup a Kerberos/SASL enabled Kafka cluster with three brokers. Everything is working except an issue I have with the topic creation. When a topic is created with a principal different from the principal of the leader broker, a znode is create under 'kafka/brokers/topics' with the name o

Topic creation fails on Kerberos/SASL enabled brokers

Hello, I have setup a Kerberos/SASL enabled Kafka cluster with three brokers. Everything is working except an issue I have with the topic creation. When a topic is created with a principal different from the principal of the leader broker, a znode is create under 'kafka/brokers/topics' with the

Re: Securing Kafka - Keystore and Truststore question

Rajini, Sure, will submit a PR shortly. Your answer is very helpful, but I think I did not put the question correctly. Pardon my ignore but I am still trying to get my ways around Kafka security. I was trying to understand, can we (Kafka Broker) just add the certificate (unsigned or signed) from

Re: Java 8 stream consumer pattern

Thanks, Matthias, I had not considered this. I will take a look at the streams use case. Thanks On Wed, May 17, 2017 at 4:58 PM, Matthias J. Sax wrote: > Did you try out Kafka Streams API instead of wrapping the consumer? It > does support Lambdas already: > > https://github.com/confluentinc/ex

Re: Securing Kafka - Keystore and Truststore question

Raghav, Yes, you can create a truststore with your customers' certificates and vice-versa. It will be best to give your CA certificate to your customers and get the CA certificate from each of your customers and add them to your broker's truststore. You can both then create additional certificates

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

I use exactly one iterator and of course I didn't close() it :-( I switched to a try-with-ressource and now my StateStore works like a charm. Huge thanks to you Eno! V. On Thu, May 18, 2017 at 10:31 AM, Eno Thereska wrote: > Hi Vincent, > > Could you share your code, the part where you write t

RE: Partition groups

To answer my own question to an extent, I guess one thing I could do is have a supplementary topic with 1/16th the partitions. You use that one for auto partition rebalancing and then subscribe explicitly to the main topic with [partition*16, partition*16 + 16) partitions. That way we can move b

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

Hi Vincent, Could you share your code, the part where you write to the state store and then delete. I'm wondering if you have iterators in your code that need to be closed(). Eno > On 16 May 2017, at 16:22, Vincent Bernardi wrote: > > Hi Eno, > Thanks for your answer. I tried sending a follow

Re: Order of punctuate() and process() in a stream processor

Hi Michal, yes, I know its beyond the scope of KIP-138, but from previous messages from Matthias I thought that he will create a new ticket, but it seems that instead he added it to KAFKA-3514. I will update that ticket with my thoughts. Thanks, Sini From: Michal Borowiecki To: users

Re: Data loss after a Kafka Broker restart scenario

Hi Jun, Thanks alot for the reply. As suggested, I tried running my application in the trunk. But I still do encounter data loss! Are there any specific configs that I need to change from its default value? What could be the reason for this? Fathima