Re: New 0.8.2 client compatibility with 0.8.1.1 during failure cases

2015-01-12 Thread Paul Pearcy
elpful to contribute them to the WIP patch in that JIRA. Currently the > patch only adds one unit test for RecordAccumulator, a few tests using the > full server would be an improvement. > > -Ewen > > On Tue, Jan 6, 2015 at 7:54 AM, Paul Pearcy wrote: > > > Hello, >

Re: Kafka 0.8.2 new producer blocking on metadata

2015-01-12 Thread Paul Pearcy
Hi Steven, Speaking only for myself, I agree with you. I think these settings/tweaks are the easiest short term way to get some proper non-blocking behavior. Long term, it seems like having a secondary queue in the client to hold raw messages until meta is available and then start blocking or dro

New 0.8.2 client compatibility with 0.8.1.1 during failure cases

2015-01-06 Thread Paul Pearcy
Hello, I have some of my own test cases very similar to the ones in ProducerFailureHandlingTest. I moved to the new producer against 0.8.1.1 and all of my test cases around disconnects fail. Moving to use 0.8.2-beta on server side things succeed. Here is an example test: - Healthy producer/serv

Re: Kafka 0.8.2 new producer blocking on metadata

2014-12-29 Thread Paul Pearcy
FYI, here is the ticket I opened for this improvement: https://issues.apache.org/jira/browse/KAFKA-1835 Feel free to add feedback on if it meets your use case and if not how things could. This should make the blocking behavior explicit as long as you know all your topics up front. Ideally a separ

Re: Kafka 0.8.2 new producer blocking on metadata

2014-12-22 Thread Paul Pearcy
> > On Wed, Dec 17, 2014 at 11:10 PM, Paul Pearcy > wrote: > > > Heya, > > Playing around with the 0.8.2-beta producer client. One of my test > cases > > is to ensure producers can deal with Kafka being down when the producer > is > > created. My test

Re: Kafka 0.8.2 new producer blocking on metadata

2014-12-21 Thread Paul Pearcy
...@jkoshy-ld.linkedin.biz%3E), it seems that 0.8.2 producer should be fully compatible with 0.8.1.1 broker. >From everything I see running in a single node config, the 0.8.2 Java producer is effectively dead after a complete disconnect to the 0.8.1.1 broker. Thanks, Paul On Sun, Dec 21, 2014 at 3:06 AM

Re: Kafka 0.8.2 new producer blocking on metadata

2014-12-21 Thread Paul Pearcy
ing requests with some exception, right? > > -Jay > > On Fri, Dec 19, 2014 at 1:55 PM, Paul Pearcy wrote: > > > > Hi Jay, > > I have implemented a wrapper around the producer to behave like I want > it > > to. Where it diverges from current 0.8.2 producer is

Re: Kafka 0.8.2 new producer blocking on metadata

2014-12-19 Thread Paul Pearcy
letion of metadata initialization will > fail immediately rather than block. > > -Jay > > > On Fri, Dec 19, 2014 at 9:32 AM, Paul Pearcy wrote: > > > > Hi Jay, > > Many thanks for the info. All that makes sense, but from an API > > standpoint when somet

Re: Kafka 0.8.2 new producer blocking on metadata

2014-12-19 Thread Paul Pearcy
ions are established, caches populated, etc) so this was > not unreasonable. > > If you want to pre-initialize the metadata to avoid blocking on the first > request you can do so by fetching the metadata using the > producer.partitionsFor(topic) api at start-up. > > -Jay >

Kafka 0.8.2 new producer blocking on metadata

2014-12-18 Thread Paul Pearcy
Hello, Playing around with the 0.8.2-beta producer client. One of my test cases is to ensure producers can deal with Kafka being down when the producer is created. My tests failed miserably because of the default blocking in the producer with regard to metadata.fetch.timeout.ms. The first line o

Kafka 0.8.2 new producer blocking on metadata

2014-12-18 Thread Paul Pearcy
Heya, Playing around with the 0.8.2-beta producer client. One of my test cases is to ensure producers can deal with Kafka being down when the producer is created. My tests failed miserably because of the default blocking in the producer with regard to metadata.fetch.timeout.ms. The first line