Broker failure causes some producers to hang for about 15 minutes

2015-09-21 Thread Jamie X
We are using Kafka 0.8.2.1 and the old producer. When there is an issue with a broker machine (we are not completely sure what the issue is, though it generally looks like host down), sometimes it takes some of the producers about 15 minutes to time out. Our producer settings are request.required.a

Re: Compression and batching

2015-05-13 Thread Jamie X
at 9:57 AM, Jamie X wrote: > (sorry if this messes up the mailing list, I didn't seem to get replies in > my inbox) > > Jiangjie, I am indeed using the old producer, and on sync mode. > > > Notice that the old producer uses number of messages as batch limitation >

Re: Compression and batching

2015-05-13 Thread Jamie X
regardless of whether it would exceed a size limit? If so that may explain things. Thanks, Jamie On Tue, May 12, 2015 at 4:40 PM, Jamie X wrote: > Hi, > > I'm wondering when you call kafka.javaapi.Producer.send() with a list of > messages, and also have compression on (snappy in

Compression and batching

2015-05-12 Thread Jamie X
Hi, I'm wondering when you call kafka.javaapi.Producer.send() with a list of messages, and also have compression on (snappy in this case), how does it decide how many messages to put together as one? The reason I'm asking is that even though my messages are only 70kb uncompressed, the broker comp