[
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805446#comment-13805446
]
Tejas Patil commented on KAFKA-1012:
------------------------------------
Hi Joe,
>> .... something like Sriram was saying
I think that Srirams' comment was more towards software engineering aspect:
suggesting for a better interface design.
>> "Isn't this something that has to be plugable"
Phase #1 had introduced "OffsetCommit" and "OffsetFetch" messages by which
consumers could send (and retrieve) their offsets to brokers *only* and let
brokers decide where to store the offsets. In the current proposal [0],
'OffsetManager' abstracts things at the broker side. Brokers can write stuff to
ZK or a special topic in kafka or a database, this is where we are trying to
achieve pluggability.
>> we could have some REST service option too
Interesting. In the Phase #1 proposal [1], there is short mention of not
supporting "pluggable storage" at client side (see section "Integration With
Clients"). Although its not trivial, I think that it can be achieved. It would
end up making client code more complicated and users (consumers) would need
additional setup for handling REST requests (I assumed that the REST requests
would be handled by some other entity and not the broker).
Thanks for your comments. There is lot of work pending in this jira but I can't
invest time in coming weeks... but please let the comments coming in !!
[0] :
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
[1] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> Implement an Offset Manager and hook offset requests to it
> ----------------------------------------------------------
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets
> from brokers. Currently, consumers are not using this API and directly
> talking with Zookeeper.
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing,
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API
--
This message was sent by Atlassian JIRA
(v6.1#6144)
