[
https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529617#comment-13529617
]
Jay Kreps commented on KAFKA-657:
---------------------------------
The existing API describes the offset ranges contained in log segments on the
server. It is poorly named and we should really rename it to something like
LogMetadataRequest and we should really generalize it a bit to include things
other than segment offset beginnings. The intended use case for the existing
API is for new consumers when they consume for the first time in an existing
stream. When they first start consuming they have no position in the log to
read from (or to save out using your new api). They want to start consuming,
but to start consuming they need a valid offset to start at. What offsets are
valid depends on what is available on the server, so they need to be able to
ask the server "what offset ranges do you have" and then they could chose to
start consuming either at the beginning or end of that (or somewhere in the
middle). Your api on the other hand answers the question "what is the latest
offset I have 'committed' (i.e. recorded as consumed)." This would be used when
a restart or rebalancing of the consumers occurs. Hope that makes sense? We
could rename the existing API as part of this change to avoid the muddle.
> Add an API to commit offsets
> ----------------------------
>
> Key: KAFKA-657
> URL: https://issues.apache.org/jira/browse/KAFKA-657
> Project: Kafka
> Issue Type: New Feature
> Reporter: Jay Kreps
> Labels: project
> Attachments: KAFKA-657v1.patch
>
>
> Currently the consumer directly writes their offsets to zookeeper. Two
> problems with this: (1) This is a poor use of zookeeper, and we need to
> replace it with a more scalable offset store, and (2) it makes it hard to
> carry over to clients in other languages. A first step towards accomplishing
> that is to add a proper Kafka API for committing offsets. The initial version
> of this would just write to zookeeper as we do today, but in the future we
> would then have the option of changing this.
> This api likely needs to take a sequence of
> consumer-group/topic/partition/offset entries and commit them all.
> It would be good to do a wiki design on how this would work and consensus on
> that first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
