Hi Zakelly, Adding such "rich" constructs is fair point, we can keep things internal. I've a rough feeling what that would mean and we can discuss the details on a PR.
> BTW, I'm curious if there is a solution for implementing a Source V2 based on `InputFormat`. Otherwise if the `InputFormat` is only for V1 then why is it not deprecated. Well, SourceBuilder [1] uses a deprecated legacy StreamSource [2]. That triggers the who issue. In short instead of StreamSource one must use Source [3]. I've seem some suspicious deprecated warnings on sink side related OutputFormatSinkFunction but not yet gone that far... [1] https://github.com/apache/flink/blob/master/flink-libraries/flink-state-processing-api/src/main/java/org/apache/flink/state/api/input/SourceBuilder.java [2] https://github.com/apache/flink/blob/a4e3084ab9220b02bc3eefc70ea372251aec7f07/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/StreamSource.java [3] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/Source.java BR, G On Fri, May 30, 2025 at 5:33 AM Zakelly Lan <zakelly....@gmail.com> wrote: > Hi Gabor, > > To be more specific, I'm thinking of introducing `RichSourceReaderContext`, > and having a `RichSourceReaderFactory` creating `SourceReader` on top of > the rich context. Or we still use the `SourceReaderFactory` but do type > conversion from `SourceReaderContext` to `RichSourceReaderContext` in > implementation of state processing source's `createReader`. Well this is > not a big change, WDTY? > > BTW, I'm curious if there is a solution for implementing a Source V2 based > on `InputFormat`. Otherwise if the `InputFormat` is only for V1 then why is > it not deprecated. > > > Best, > Zakelly > > On Thu, May 29, 2025 at 10:18 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > > > Hi Zakelly, > > > > Before we go into any details I can say that if we can expose the less > the > > better so agree with you on the direction. > > > > On the technical level not yet understand what exactly you're suggesting. > > `RichInputFormat` is a \@Public API > > where we expose `RuntimeContext` now on V1. I would like to understand > the > > motivation what would be the difference > > between `SourceReaderContext.getRuntimeContext` vs > > `V2RichInputFormat.getRuntimeContext`. > > Or you mean something like `RichSourceReaderContext`? > > > > If you can be more explicit on how you imagine the other path with some > new > > interface/class names, > > call flow or anything we can pursue that. > > > > On coding level now I'm sitting on ~3-4K lines of code but there are some > > quite things needs to be done. > > The whole code is depending on `SourceReaderContext.getRuntimeContext` > but > > I think when it's stable > > and there is another approach how to expose `RuntimeContext` then it's > only > > matter of time to make that > > adjustments. Hope that we're not shooting for another 10k change :) > > > > BR, > > G > > > > > > On Thu, May 29, 2025 at 10:23 AM Zakelly Lan <zakelly....@gmail.com> > > wrote: > > > > > Hi Gabor, > > > > > > Good point for the migration. > > > > > > I took a brief look. I thought the `RuntimeContext` is too powerful and > > > might not be suitable to expose on \@Public API. Is it possible to > > > introduce another 'rich' code path just like the difference between > > > `RichInputFormat` and `InputFormat`, and keep that internal? WDYT > > > > > > > > > Best, > > > Zakelly > > > > > > On Thu, May 22, 2025 at 6:30 PM Gabor Somogyi < > gabor.g.somo...@gmail.com > > > > > > wrote: > > > > > > > Hi All, > > > > > > > > I've just had a look at it how it would be possible to migrate state > > > > processor API from source v1 to source v2 API. > > > > > > > > The whole concept in the state processor API actual implementation is > > > based > > > > on to get > > > > RuntimeContext from RichInputFormat [1] which then allows us to > access > > > > state backend, > > > > keyed state store, etc... > > > > > > > > Since RuntimeContext is not exposed on source v2 API the question is > > > > obvious. > > > > Does it make sense to expose it on SourceReaderContext [2] ? > > > > > > > > If somebody has better idea then please share. > > > > > > > > [1] > > > > > > > > > > > > > > https://github.com/apache/flink/blob/4c77daaf3554446e67f6b75ac18d54da84208b9a/flink-libraries/flink-state-processing-api/src/main/java/org/apache/flink/state/api/input/KeyedStateInputFormat.java#L156 > > > > [2] > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SourceReaderContext.java > > > > > > > > BR, > > > > G > > > > > > > > > >
