I'd suggest using the new consumer instead of the old consumer. We've
refined the implementation such that even with auto-commit you should get
at least once processing in the worst case (and when there aren't failures,
exactly once). The 0.10.0.0 release should get all of these semantics right.

-Ewen

On Mon, Jul 11, 2016 at 7:05 AM, Gerard Klijs <gerard.kl...@dizzit.com>
wrote:

> You could set the auto.commit.interval.ms to a lower value, in your
> example
> it is 10 seconds, which can be a lot of messages. I don't really see how it
> could be prevented any further, since offset's can only committed by
> consumer to the partitions they are assigned to. I do believe there is some
> work in progress in which the assigned of partitions to consumers is
> somewhat sticky.
> In that case when a consumer has been assigned the same partitions after
> the rebalance as it has had before, and then it should not be necessary to
> consume the same data again in those partitions.
>
> On Mon, Jul 11, 2016 at 3:18 PM Michael Luban <mluban....@gmail.com>
> wrote:
>
> > Using the 0.8.2.1 client.
> >
> > Is it possible to statistically minimize the possibility of duplication
> in
> > this scenario or has this behavior been corrected in a later client
> > version?  Or is the test flawed?
> >
> > https://gist.github.com/mluban/03a5c0d9221182e6ddbc37189c4d3eb0
> >
>



-- 
Thanks,
Ewen

Reply via email to