[
https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534122#comment-13534122
]
Jay Kreps commented on KAFKA-657:
---------------------------------
This looks great!
Three minor things:
1. Can you change the logging for the common case to debug? Our logging policy
is that you should be able to run in INFO and have all messages be things you
need to know.
2. Can you handle any exceptions from ZK and send back an UnknownException
3. Can you remove the checks on topic/partition validity?
(3) is maybe more controversial. Here is my rationale. First ZK is a huge
bottleneck so adding two more zk round-trips will be a problem. Second we
actually have two use cases for allowing the user to store offsets for
non-existant topics or partitions. The first use case is that if you are doing
mirroring between two clusters in different data centers (a common case) it
probably makes sense to store the offsets in whatever data center the mirroring
process runs in. However there is no requirement that the two clusters have the
same partitioning. The second use case is probably specific only to our usage,
we have several systems that produce "offset"-like markers and being able to
commit these all together to mark a single "point in consumption time" across
all systems is nice.
> Add an API to commit offsets
> ----------------------------
>
> Key: KAFKA-657
> URL: https://issues.apache.org/jira/browse/KAFKA-657
> Project: Kafka
> Issue Type: New Feature
> Reporter: Jay Kreps
> Labels: project
> Attachments: KAFKA-657v1.patch, KAFKA-657v2.patch
>
>
> Currently the consumer directly writes their offsets to zookeeper. Two
> problems with this: (1) This is a poor use of zookeeper, and we need to
> replace it with a more scalable offset store, and (2) it makes it hard to
> carry over to clients in other languages. A first step towards accomplishing
> that is to add a proper Kafka API for committing offsets. The initial version
> of this would just write to zookeeper as we do today, but in the future we
> would then have the option of changing this.
> This api likely needs to take a sequence of
> consumer-group/topic/partition/offset entries and commit them all.
> It would be good to do a wiki design on how this would work and consensus on
> that first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
