[
https://issues.apache.org/jira/browse/KAFKA-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833068#comment-13833068
]
Imran Rashid commented on KAFKA-1144:
-------------------------------------
Hi Jun,
yes, that will do the trick. I think it will be fine w/out the conditional
update -- it just means a bigger re-read if the consumer dies just after
rebalancing. Pretty rare, and not a big deal even if it does occur.
glad to hear KAFKA-1000 will make it into 0.8.1. Let me know if it would help
at all for me to update this patch, either to remove the conditional update, or
to change the method signature to match the new version.
thanks
> commitOffsets can be passed the offsets to commit
> -------------------------------------------------
>
> Key: KAFKA-1144
> URL: https://issues.apache.org/jira/browse/KAFKA-1144
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Affects Versions: 0.8
> Reporter: Imran Rashid
> Assignee: Neha Narkhede
> Attachments:
> 0001-allow-committing-of-arbitrary-offsets-to-facilitate-.patch,
> 0002-add-protection-against-backward-commits.patch,
> 0003-dont-do-conditional-update-check-if-the-path-doesnt-.patch
>
>
> This adds another version of commitOffsets that takes the offsets to commit
> as a parameter.
> Without this change, getting correct user code is very hard. Despite kafka's
> at-least-once guarantees, most user code doesn't actually have that
> guarantee, and is almost certainly wrong if doing batch processing. Getting
> it right requires some very careful synchronization between all consumer
> threads, which is both:
> 1) painful to get right
> 2) slow b/c of the need to stop all workers during a commit.
> This small change simplifies a lot of this. This was discussed extensively on
> the user mailing list, on the thread "are kafka consumer apps guaranteed to
> see msgs at least once?"
> You can also see an example implementation of a user api which makes use of
> this, to get proper at-least-once guarantees by user code, even for batches:
> https://github.com/quantifind/kafka-utils/pull/1
> I'm open to any suggestions on how to add unit tests for this.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
