Hey Nick, I haven't been following the new IQv2 work very closely so take this with a grain of salt, but as far as I'm aware there's no such thing as "position files" -- the Position is just an in-memory object and is related to a user's query against the state store, whereas a checkpoint file reflects the current state of the store ie how much of the changelog it contains.
In other words while these might look like they do similar things, the actual usage and implementation of Positions vs checkpoint files is pretty much unrelated. So I don't think it would sense for Streams to try and consolidate these or replace one with another. Hope this answers your question, and I'll ping John to make sure I'm not misleading you regarding the usage/intention of Positions Sophie On Fri, Nov 11, 2022 at 6:48 AM Nick Telford <nick.telf...@gmail.com> wrote: > Hi everyone, > > I'm trying to understand how StateStores work internally for some changes > that I plan to propose, and I'd like some clarification around checkpoint > files and position files. > > It appears as though position files are relatively new, and were created as > part of the IQv2 initiative, as a means to track the position of the local > state store so that reads could be bound by particular positions? > > Checkpoint files look much older, and are managed by the Task itself > (actually, ProcessorStateManager). It looks like this is used exclusively > for determining a) whether to restore a store, and b) which offsets to > restore from? > > If I've understood the above correctly, is there any scope to potentially > replace checkpoint files with StateStore#position()? > > Regards, > > Nick >
