[
https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534162#comment-13534162
]
Jay Kreps commented on KAFKA-657:
---------------------------------
I think that actually covers an orthogonal problem right?
1. Checking topic/partition covers bugs in the client impl that set the wrong
values.
2. Check and set catches a bug that might lead to you clobbering your offset
due to a concurrency issue where there are two processes both trying to update
the same offset.
Originally my concern with (2) was that I wasn't sure if we could implement it
in a post-zk world. Now that we wrote up that proposal in a lot more detail I
think we can.
We wouldn't want to make the last offset mandatory because in the case that you
are manually resetting your offset to 0 (or some low number) you might not know
the previous value. But I think what you are proposing is that we could have a
current_offset field in the request, and if it is set we would only update the
offset if the current offset equals the given offset. We could make it optional
by having the value -1 indicate "don't care, clobber whatever is there".
The question is, what is the use case for this? Our approach to the scala
client has been to ensure mutual exclusion for the consumer processes, at which
point this basically can't happen. I wonder how an alternative client
implementation could make use of it? It would be good to work that out before
including it.
> Add an API to commit offsets
> ----------------------------
>
> Key: KAFKA-657
> URL: https://issues.apache.org/jira/browse/KAFKA-657
> Project: Kafka
> Issue Type: New Feature
> Reporter: Jay Kreps
> Labels: project
> Attachments: KAFKA-657v1.patch, KAFKA-657v2.patch
>
>
> Currently the consumer directly writes their offsets to zookeeper. Two
> problems with this: (1) This is a poor use of zookeeper, and we need to
> replace it with a more scalable offset store, and (2) it makes it hard to
> carry over to clients in other languages. A first step towards accomplishing
> that is to add a proper Kafka API for committing offsets. The initial version
> of this would just write to zookeeper as we do today, but in the future we
> would then have the option of changing this.
> This api likely needs to take a sequence of
> consumer-group/topic/partition/offset entries and commit them all.
> It would be good to do a wiki design on how this would work and consensus on
> that first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
