Re: Producer.send questions

2013-08-24 Thread Jason Rosenberg
Jun, There are several others I've seen that I would have thought would be retryable (possibly after an exponential backoff delay). I'm curious about: BrokerNotAvailableException FailedToSendMessageException QueueFullException (happens if producerType is 'async') KafkaException (this seems to wr

Re: Producer.send questions

2013-08-24 Thread Jun Rao
You don't need to restart the producer. The producer currently handles all error/exceptions by refreshing the metadata and retrying. If it fails all retries, it throws a FailedToSendMessageException to the caller (in sync mode). The original cause is not included in this exception. We have thought

Re: Producer.send questions

2013-08-24 Thread Jason Rosenberg
Jun, Thanks, this is helpful. So, can QueueFullException occur in either sync or async mode (or just async mode)? If there's a MessageSizeTooLargeException, is there any visibility of this to the caller? Or will it just be a FailedToSendMessageException. I gathered from one of your previous re

Consumer throughput imbalance

2013-08-24 Thread Ian Friedman
Hey guys! We recently deployed our kafka data pipeline application over the weekend and it is working out quite well once we ironed out all the issues. There is one behavior that we've noticed that is mildly troubling, though not a deal breaker. We're using a single topic with many partitions (1

Re: Producer.send questions

2013-08-24 Thread Neha Narkhede
>> So, can QueueFullException occur in either sync or async mode (or just async mode)? QueueFullException can only occur in async mode since there is no queue in sync mode. >> If there's a MessageSizeTooLargeException, is there any visibility of thisto the caller? The kafka producer should not r

Re: Producer.send questions

2013-08-24 Thread Jason Rosenberg
Thanks Neha, On Sat, Aug 24, 2013 at 10:06 AM, Neha Narkhede wrote: > > >> I gathered from one of your previous responses, that a > MessageSizeTooLargeException > can be rectified with a smaller batch size. > If so, does that imply that the message size limit is measured on the > broker by the cu

Re: Producer.send questions

2013-08-24 Thread Neha Narkhede
>> Ok, but perhaps the producer will handle something like this in the future? Yes, I think we need a JIRA for this. >> UnretryableFailedToSendMessageException (wraps the root cause) NoMoreRetriesFailedToSendMessageException (wraps the root cause, from the final attempt) Something like this make