[
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754024#comment-13754024
]
Tejas Patil commented on KAFKA-1012:
------------------------------------
> zkClient, logManager are implementation specific
Agree. Would get rid of startup from the interface and have these specific
things passed to 'createOffsetManager'
> triggerLoadOffsets should also not be part of the interface
We had thought of making this completely inbuilt in the DefaultOffsetManager.
This is what was thought:
1. Have the startup method of offset manager do this bootstrap using an async
thread. It was observed that the log manager was not up immediately. It
schedules some threads to load the logs and we need to have a latch waiting for
LogManager to init logs. Even if this is in place, there is a need for a
mechanism is needed to tell offset manager to load logs.
2. Do the loading when a fetch requet comes. This would delay the request
response this would hit perf.
I need to put in more thinking to come up with a way to have loading performed
behind the hood.
As for the name, 'triggerLoadOffsets' can be renamed to 'syncOffsetsFromLog' if
that sounds better.
> make the OffsetManager completely pluggable without needing to rebuild the
> kafka jar
This would be a good thing to have. For now, much work is needed in getting the
feature tested and getting rid of embedded producer. This would be somewhere
later in the roadmap.
Those were some good suggestions for improving the design. Thanks [~sriramsub]
:)
> Implement an Offset Manager and hook offset requests to it
> ----------------------------------------------------------
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets
> from brokers. Currently, consumers are not using this API and directly
> talking with Zookeeper.
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing,
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
