Hi Raul! Concerning the source connector and position marker: Great idea! The FlinkKafkaConsumer uses pretty much the same trick - the offset-per-partition is used to filter during replays.
Greetings, Stephan On Tue, Apr 5, 2016 at 2:24 AM, Raul Kripalani <ra...@apache.org> wrote: > On Mon, Apr 4, 2016 at 3:54 PM, Stephan Ewen <se...@apache.org> wrote: > > > > > - Sounds like the having Ignite for snapshots should work pretty much > > out > > of the box (via the IGFS) > > - The source and sink connector sounds like the next logical step. Does > > Ignite have a notion of stream partitions and offsets, to build a > > consistent replay around? This should probably have its dedicated issue > and > > discussion thread. > > > > - For Ignite as an execution backend - I am not sure how relevant and > > feasible that is. Many DataStream API features make use of the specific > > Flink runtime. For streaming, the runtime is not as decoupled as for > > batch. > > - I think the parameter server integration would not be part of the > > Flink > > codebase - this is a pretty application specific thing that should be its > > own project and it is actually not tightly coupled to Flink. > > > Danke, Stephan! I think I'll start with the sink/source connector – reusing > what's already been committed to our codebase. > > With regards to source replayability, I plan to integrate Ignite Continuous > Queries as a source. If the user's data objects contain an indexed > ascending numeric or datetime field, we could use such a field as a > "position marker" by launching the query with the appropriate WHERE filter > when a replay is demanded. > > Do you have similar use cases with existing connectors? > > Cheers, > > *Raúl Kripalani* > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and > Messaging Engineer > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani > Blog: raul.io > <http://raul.io/?utm_source=email&utm_medium=email&utm_campaign=apache> | > twitter: @raulvk <https://twitter.com/raulvk> >
