[
https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529583#comment-13529583
]
Jay Kreps commented on KAFKA-657:
---------------------------------
This looks great. To confirm, the final format for the commit response is
group [topic [partition offset]]
I think logically there are two phases of work around fixing offset management
1. Add the API and convert the consumer to use it
a. CommitOffsetRequest/Response (to save your position)
b. FetchOffsetRequest/Response (to read back a saved position)
c. Integration into the consumer (using the new api in the scala client)
d. Unit test coverage for these (say in kafka.integration.PrimitiveApiTest)
2. Move offsets out of zookeeper, since zookeeper doesn't scale well for writes
It would be nice to do (1) more or less together, and if we do it right (2) can
be a follow-up item and need not be done by you unless you want to. We can
definitely break (1) into successive patches if that is helpful to keep the
individual changes small--I am happy to take what you have now if you are up
for finishing the other items in (1). I would like to get people to brainstorm
a little on (2) in parallel as it could potentially have some impact on (1). We
have some time to fiddle with the API if we think of improvements before it
would be released and we would have to start versioning changes, though, so we
probably don't need to block on that.
So let me know if you are up for the rest of the items in (1)
> Add an API to commit offsets
> ----------------------------
>
> Key: KAFKA-657
> URL: https://issues.apache.org/jira/browse/KAFKA-657
> Project: Kafka
> Issue Type: New Feature
> Reporter: Jay Kreps
> Labels: project
> Attachments: KAFKA-657v1.patch
>
>
> Currently the consumer directly writes their offsets to zookeeper. Two
> problems with this: (1) This is a poor use of zookeeper, and we need to
> replace it with a more scalable offset store, and (2) it makes it hard to
> carry over to clients in other languages. A first step towards accomplishing
> that is to add a proper Kafka API for committing offsets. The initial version
> of this would just write to zookeeper as we do today, but in the future we
> would then have the option of changing this.
> This api likely needs to take a sequence of
> consumer-group/topic/partition/offset entries and commit them all.
> It would be good to do a wiki design on how this would work and consensus on
> that first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
