On Mon, Apr 4, 2016 at 3:54 PM, Stephan Ewen <se...@apache.org> wrote:
> > - Sounds like the having Ignite for snapshots should work pretty much > out > of the box (via the IGFS) > - The source and sink connector sounds like the next logical step. Does > Ignite have a notion of stream partitions and offsets, to build a > consistent replay around? This should probably have its dedicated issue and > discussion thread. > > - For Ignite as an execution backend - I am not sure how relevant and > feasible that is. Many DataStream API features make use of the specific > Flink runtime. For streaming, the runtime is not as decoupled as for > batch. > - I think the parameter server integration would not be part of the > Flink > codebase - this is a pretty application specific thing that should be its > own project and it is actually not tightly coupled to Flink. Danke, Stephan! I think I'll start with the sink/source connector – reusing what's already been committed to our codebase. With regards to source replayability, I plan to integrate Ignite Continuous Queries as a source. If the user's data objects contain an indexed ascending numeric or datetime field, we could use such a field as a "position marker" by launching the query with the appropriate WHERE filter when a replay is demanded. Do you have similar use cases with existing connectors? Cheers, *Raúl Kripalani* PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and Messaging Engineer http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani Blog: raul.io <http://raul.io/?utm_source=email&utm_medium=email&utm_campaign=apache> | twitter: @raulvk <https://twitter.com/raulvk>
