Re: Too Many Open Files

2016-07-30 Thread Kessiler Rodrigues
I have changed it a bit. I have 10 brokers and 20k topics with 1 partition each. I looked at the kaka’s logs dir and I only have 3318 files. I’m doing some tests to see how many topics/partitions I can have, but it is throwing too many files once it hits 15k topics.. Any thoughts? > On Jul

Re: Kafka 0.9.0.1 failing on new leader election

2016-07-30 Thread Sean Morris (semorris)
Kafka 0.9.0.1 Zookeeper 3.4.6 Zkclient 0.7 I have verified I only have one zkclient.jar in my class path. Thanks, Sean On 7/29/16, 9:35 PM, "Gwen Shapira" wrote: >you know, I ran into those null pointer exceptions when I accidentally >tested Kafka with mismatching version of zkclient. > >Ca

Re: Reactive Kafka performance

2016-07-30 Thread Guozhang Wang
Hello David, I'd love to learn details about the "flexibility" of Reactive Kafka compared with KStreams in order to see if KStreams can improve on that end. Would you like to elaborate a bit more on that end? Guozhang On Thu, Jul 28, 2016 at 12:16 PM, David Garcia wrote: > Our team is evaluat

Re: Reactive Kafka performance

2016-07-30 Thread David Garcia
Hey Guozhang, our basic road block is asynchronous processing (this is actually related to my previous post about asynchronous processing). Here is the simplest use-case: The streaming job receives messages. Each message is a user-event and needs to randomly look up that user’s history (for 1

Connection to broker which lost Zk connection

2016-07-30 Thread Krzysztof Nawara
Hello! I've got the following question: Let's assume broker A is leader for partition P1, to which my producer is sending data. At one point A loses its connection to Zk, but my producer can communicate with A without problems. Will my messages be lost? What'll trigger switching to the new lead

RAID 10

2016-07-30 Thread Krzysztof Nawara
Hello, Could you share your experience with using RAID 10 for brokers? I'm especially interested in whether you can keep broker fully operational during RAID rebuild or it falls behind so much that it must go through lengthy resynchronization process anyway? Thanks, Krzysiek

Re: [kafka-clients] [VOTE] 0.10.0.1 RC0

2016-07-30 Thread Ismael Juma
Hi Dana, Thanks for testing releases so promptly. Very much appreciated! It's funny, Ewen had suggested something similar with regards to the release notes a couple of days ago. We now have a Python script for generating the release notes: https://github.com/apache/kafka/blob/trunk/release_notes

Re: [VOTE] 0.10.0.1 RC0

2016-07-30 Thread Ismael Juma
Hi Harsha, I agree that we should fix KAFKA-3950 and include it in the 0.10.0 branch. I worked with Manikumar on the PR to improve the tests and to fix a bug for the case where the subscription pattern matches internal topics and `exclude.internal.topics` is true (the default). We should be able t