Re: Kafka consumer transactional support

2014-11-12 Thread Falabella, Anthony
I didn't realize there was a commitOffset() method on the high level consumer (the code is abstracted by the Spring Integration classes). Yes, this actually suits my needs and I was able to get it to work for my use case. Thank you very much - that was extremely helpful. In case it's of any use

Re: Kafka consumer transactional support

2014-11-12 Thread Jun Rao
Yes, the new consumer api will solve your probably better. Before that's ready, another option is to use the commitOffset() api in the high level consumer. It doesn't take any offset though. So, to prevent message loss during consumer failure, you will need to make sure all iterated messages are fu

Re: Kafka consumer transactional support

2014-11-12 Thread Falabella, Anthony
Hi Jun, Thanks for taking a look at my issue and also for updating the future release plan Wiki page. My use case is to use Kafka as if it were a JMS provider (messaging use case). I'm currently using Kafka 0.8.1.1 with Java and specifically the Spring Integration Kafka Inbound Channel Adapter

- Re: Kafka consumer transactional support

2014-11-12 Thread Falabella, Anthony
Hi Jun, Thanks for taking a look at my issue and also for updating the future release plan Wiki page. My use case is to use Kafka as if it were a JMS provider (messaging use case). I'm currently using Kafka 0.8.1.1 with Java and specifically the Spring Integration Kafka Inbound Channel Adapter

Re: Kafka consumer transactional support

2014-11-11 Thread Jun Rao
The transactional work is currently just a prototype. The release wiki is a bit outdated and I just updated it. Do you have a particular transactional use case in mind? Thanks, Jun On Mon, Nov 10, 2014 at 3:23 PM, Falabella, Anthony < anthony.falabe...@citi.com> wrote: > I've tried to search a

Kafka consumer transactional support

2014-11-10 Thread Falabella, Anthony
I've tried to search a bit for what the current status is for Kafka supporting consumer transactions. These links look to be the best indication of where things stand with that: http://search-hadoop.com/m/4TaT4xeqNq1 https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Ka