[
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753726#comment-13753726
]
Sriram Subramanian commented on KAFKA-1012:
-------------------------------------------
Looks promising. Took a look at OffsetManager. The interface for OffsetManager
needs more thinking. The zkClient, logManager are implementation specific.
triggerLoadOffsets should also not be part of the interface. Imagine if someone
needs to extend a new OffsetManager and they need to add a new k/v store
client. They would have to end up modifying the interface to add their client.
All this should at the minimum be part of the constructor. Even better would be
if we can make the OffsetManager completely pluggable without needing to
rebuild the kafka jar. Removing triggerLoadOffsets is going to need some amount
of thinking though. At a minimum we need to think of a better name.
trait OffsetManager extends Logging {
protected var zkClient: ZkClient = null
def startup(zkClient: ZkClient, logManager: LogManager = null)
def getOffset(key: GroupTopicPartition): OffsetMetadataAndError
def putOffset(key: GroupTopicPartition, offset: OffsetAndMetadata)
def triggerLoadOffsets(partition: Int)
def shutdown()
}
> Implement an Offset Manager and hook offset requests to it
> ----------------------------------------------------------
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets
> from brokers. Currently, consumers are not using this API and directly
> talking with Zookeeper.
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing,
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
