Re: data loss on replicated topic

2014-03-28 Thread Oliver Dain
s sounds like a bug. Do >you see any other error/warn in broker log around the time the data is >lost? > >Thanks, > >Jun > > >On Thu, Mar 27, 2014 at 10:52 AM, Oliver Dain >wrote: > >> Hi Neha, >> >> Thanks for the reply. I do not see the ³No broker

Re: data loss on replicated topic

2014-03-27 Thread Oliver Dain
alive for ... There's potential data loss." > >Thanks, >Neha > > >On Wed, Mar 26, 2014 at 12:05 PM, Oliver Dain >wrote: > >> I just saw https://issues.apache.org/jira/browse/KAFKA-1193 which seems >> like it could be the cause of this. Does that sound ri

Re: data loss on replicated topic

2014-03-26 Thread Oliver Dain
I just saw https://issues.apache.org/jira/browse/KAFKA-1193 which seems like it could be the cause of this. Does that sound right? Is there a patch we can test? Any date/time when this is expected to be fixed? From: New User mailto:od...@3cinteractive.com>> Date: Wednesday, March 26, 2014 at 11:

data loss on replicated topic

2014-03-26 Thread Oliver Dain
My company currently testing Kafka for throughput and fault tolerance. We've set up a cluster of 5 Kafka brokers and are publishing to a topic with replication factor 3 and 100 partitions. We are publishing with request.required.acks == -1 (e.g. All ISR replicas must ACK before the message is c

Re: New Producer Public API

2014-01-31 Thread Oliver Dain
register the callback with the future. This is not nearly as important as I’d thought given the ability to register a callback - just a preference. On 1/31/14, 9:33 AM, "Oliver Dain" wrote: >Hey all, > >I¹m excited about having a new Producer API, and I really like the id

Re: New Producer Public API

2014-01-31 Thread Oliver Dain
I wanted to suggest an alternative to the serialization issue. As I understand it, the concern is that if the user is responsible for serialization it becomes difficult for them to compute the partition as the plugin that computes the partition would be called with byte[] forcing the user to de-ser

Re: New Producer Public API

2014-01-31 Thread Oliver Dain
Hey all, I¹m excited about having a new Producer API, and I really like the idea of removing the distinction between a synchronous and asynchronous producer. The one comment I have about the current API is that it¹s hard to write truly asynchronous code with the type of future returned by the send

Re: Writing unit tests for Kafka code

2013-12-27 Thread Oliver Dain
I finally got something working. It was a bit funky, so I figure I¹ll post it here in hopes that it will help someone else. First, you need to include the test stuff from Kafka. If using Maven, this does the trick: org.apache.kafka kafka_2.10 test test I used Apache Curator to get

Re: Writing unit tests for Kafka code

2013-12-27 Thread Oliver Dain
Hey Joe, Thanks for the suggestion. The code looks pretty cool, but it¹s much heavier weight than what I¹m looking for. I can spin up ZooKeeper and Kafka on my local box, but I was hoping to find a way to run them embedded so I can create unit tests (as opposed to integration tests which would use

Re: Writing unit tests for Kafka code

2013-12-27 Thread Oliver Dain
Hi Jun, Thanks for getting back to me. Unfortunately, I can't seem to get KafkaServerTestHarness to work for me. I've looked around the internet for examples but can't find any in Java. My Java compiler seems to think that KafkaServerTestHarness isn't a class so it won't let me instantiate one

Re: Writing unit tests for Kafka code

2013-12-18 Thread Oliver Dain
Found some of the other references I'd used that also don't quite work: https://gist.github.com/fjavieralba/7930018 https://gist.github.com/mardambey/2650743 I know, the obvious question is what didn't work. I'm afraid I no longer recall. I took each example, tried it, got errors, tried to fix t

Writing unit tests for Kafka code

2013-12-18 Thread Oliver Dain
I'm writing some Kafka client code and I'd like to unit test it. I've found some resources that say this is possible. For example: http://ransilberman.wordpress.com/2013/07/19/how-to-unit-test-kafka/ but all the information I've found seems a little bit incorrect. For example, the above link us