Re: Patterns for message failure handling with Kafka

2014-01-21 Thread Jason Rosenberg
So, I think there are 2 different types of errors you mention. The first is data-dependent (e.g. it's corrupt or some such). So, there's no reason to block consumption of other messages that are likely to be successful, while the data-dependent one won't fix itself no matter times you retry. So,

Patterns for message failure handling with Kafka

2014-01-21 Thread Jim
I'm looking at message delivery patterns for Kafka consumers and wanted to get people's thoughts on the following problem: The objective is to ensure processing of individual messages with as much certainty as possible for "at least once guarantees". I'm looking to have a kafka consumer pull n me