[
https://issues.apache.org/jira/browse/KAFKA-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewen Cheslack-Postava updated KAFKA-3820:
-----------------------------------------
Labels: needs-kip (was: )
> Provide utilities for tracking source offsets
> ---------------------------------------------
>
> Key: KAFKA-3820
> URL: https://issues.apache.org/jira/browse/KAFKA-3820
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect
> Reporter: Ewen Cheslack-Postava
> Assignee: Liquan Pei
> Priority: Minor
> Labels: needs-kip
>
> OffsetStorageReader does not (and is not expected to) be immediately updated
> when a SourceRecord is returned from poll(). However, this can be a bit
> confusing to connector developers as they may return that data, then expect a
> subsequent read from OffsetStorageReader should match that. In other words,
> rather than tracking which offset they are at themselves in variables
> maintained by the task implementation, the connector developer expected
> OffsetStorageReader to do this for them.
> Part of the confusion comes from the fact that data is sent asynchronously
> after returned from poll(), which explains the semantics we have. However, it
> does also mean many connectors have similarly structured code where they keep
> track of the current offset themselves. It might be nice to provide some
> utilities, probably through the Context object, to get the last returned
> offset for each source partition being processed by a task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
