Re: Idea how to ensure exactly once message deliver without external storage

2015-07-16 Thread Jiangjie Qin
Hey Stefan, I think if you follow the one-commit-per-message approach, you will be able to achieve exact once semantic. However this would be very expensive and also everything has to be synchronous in order to make it work. In that sense, de-duplication on client side seems a more favorable opti

Re: Idea how to ensure exactly once message deliver without external storage

2015-07-16 Thread Jason Gustafson
Hey Stefan, I only see a commit in the failure case. Were you planning to use auto-commits otherwise? You'd probably want to handle all commits directly or you'd always be left guessing. But even if you did, I think the main problem is that your process could fail before a needed commit is sent to

Idea how to ensure exactly once message deliver without external storage

2015-07-16 Thread Stefan Miklosovic
Hi, In the old consumer, I have got just a simple stream of messages, one by one and if I detected something was wrong, I would destroy my consumer immediately without commit so once I restart consumer, I will get the same messages once again because they will be delivered to me from the last offs