Hi Shapira, We only expose the OffsetStorageReader to connector which relies on KafkaOffsetBackingStore. The store continuesly consumes offsets from kafka so I think we can't have stale data.
Le 8 sept. 2017 06:13, "Randall Hauch" <rha...@gmail.com> a écrit : > The KIP and PR expose the OffsetStorageReader, which is already exposed to > the tasks. The OffsetStorageWriter is part of the implementation, but was > not and is not exposed thru the API. > > > On Sep 7, 2017, at 9:04 PM, Gwen Shapira <g...@confluent.io> wrote: > > > > I just re-read the code for the OffsetStorageWriter, and ran into this > > comment: > > > > * Note that this only provides write functionality. This is > > intentional to ensure stale data is > > * never read. Offset data should only be read during startup or > > reconfiguration of a task. By > > * always serving those requests by reading the values from the backing > > store, we ensure we never > > * accidentally use stale data. (One example of how this can occur: a > > task is processing input > > * partition A, writing offsets; reconfiguration causes partition A to > > be reassigned elsewhere; > > * reconfiguration causes partition A to be reassigned to this node, > > but now the offset data is out > > * of date). Since these offsets are created and managed by the > > connector itself, there's no way > > * for the offset management layer to know which keys are "owned" by > > which tasks at any given > > * time. > > > > > > I can't figure out how the KIP avoids the stale-reads problem explained > here. > > > > Can you talk me through it? I'm cancelling my vote since right now > > exposing this interface sounds risky and misleading. > > > > > > Gwen > > > > > >> On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira <g...@confluent.io> wrote: > >> > >> +1 (binding) > >> > >> Looking forward to see how connector implementations use this in > practice > >> :) > >> > >>> On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch <rha...@gmail.com> wrote: > >>> > >>> I'd like to open the vote for KIP-131: > >>> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+ > Add+access+to+OffsetStorageReader+from+SourceConnector > >>> > >>> Thanks to Florian for submitting the KIP and the implementation, and to > >>> everyone else that helped review. > >>> > >>> Best regards, > >>> > >>> Randall > >>> > >> >
