Currently, consumers must deal with ZooKeeper directly and implement the
consumer re-balancing algorithm (correctly). As this is a rather
difficult and error-prone process, the intent of this new API is to
provide an easy mechanism for storing offsets for non-Scala clients.
At least, that's how I view it. Jay and others may have insights as to
how this plays into the overall strategy of the project.
On 12/17/12 5:27 PM, Milind Parikh wrote:
Perhaps I don't understand the motivation well enough and perhaps I am
misreading the intent.
But I thought that the design principle behind kafka is for state
(from a consumer standpoint) was to be managed by consumer and not
broker. I understand that "These APIs are optional, clients can store
offsets another way if they like."
So three questions :
(a) Is this a change from the original intent of Kafka?
(b) If it is a change, why not make it such that there is no need for
clients to roll their own?
autocommit=false-> no storage of offsets
autocommit=true -> store offsets
(c) I suppose that the use case of multiple sets of consumer groups
wanting to use the offsets for different purposes could be one of the
use cases for the clients to roll their own. That corner case could be
handled through handing out a uuid for a set of consumer group to
operate against.Any other use cases for the clients to absolutely roll
their own?
Regards
Milind
On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps <jay.kr...@gmail.com
<mailto:jay.kr...@gmail.com>> wrote:
Hey Guys,
David has made a bunch of progress on the offset commit api
implementation.
Since this is a public API it would be good to do as much thinking
up-front
as possible to minimize future iterations.
It would be great if folks could do the following:
1. Read the wiki here:
https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
2. Check out the code David wrote here:
https://issues.apache.org/jira/browse/KAFKA-657
In particular our hope is that this API can act as the first step in
scaling the way we store offsets (ZK is not really very
appropriate for
this). This of course requires having some plan in mind for offset
storage.
I have written (and then after getting some initial feedback,
rewritten) a
section in the above wiki on how this might work.
If no one says anything I will be taking a slightly modified patch
that
adds this functionality on trunk as soon as David gets in a few minor
tweaks.
-Jay
