Hi! - Sounds like the having Ignite for snapshots should work pretty much out of the box (via the IGFS) - The source and sink connector sounds like the next logical step. Does Ignite have a notion of stream partitions and offsets, to build a consistent replay around? This should probably have its dedicated issue and discussion thread.
- For Ignite as an execution backend - I am not sure how relevant and feasible that is. Many DataStream API features make use of the specific Flink runtime. For streaming, the runtime is not as decoupled as for batch. - I think the parameter server integration would not be part of the Flink codebase - this is a pretty application specific thing that should be its own project and it is actually not tightly coupled to Flink. Greetings, Stephan On Mon, Apr 4, 2016 at 4:35 PM, Robert Metzger <rmetz...@apache.org> wrote: > Hi Raul, > > thanks a lot for reaching out to the Flink community. > I'm really excited to see a Flink connector in Ignite. If you feel that the > connector would be more suitable for our "connector library" feel free to > open a JIRA and open a pull request. > > Were there requests in the Ignite community to have an integration with > Flink? > > > > On Thu, Mar 31, 2016 at 5:20 PM, Saikat Maitra <saikat.mai...@gmail.com> > wrote: > > > Hi , > > > > I agree with Roman and Raul. > > https://issues.apache.org/jira/browse/IGNITE-813 allows injecting data > to > > into cache via Data Streamer. Integrating with Ignite FileSystem for > source > > and sink will allow for bidirectional connector. It will also allow > easier > > implementation for DataStream transformations over Ignite FileSystem. > > > > Regards > > Saikat > > > > On Thu, Mar 31, 2016 at 2:44 PM, Aljoscha Krettek <aljos...@apache.org> > > wrote: > > > > > Hi, > > > it should already be possible to use the Ignite FileSystem to store > state > > > since we just use the HDFS FileSystem interface for that. Of course, > one > > > would have to properly set up the jars and paths and everything for > Flink > > > to pick up the IGFS classes. > > > > > > Cheers, > > > Aljoscha > > > > > > On Wed, 30 Mar 2016 at 16:50 Raul Kripalani <ra...@apache.org> wrote: > > > > > > > On Wed, Mar 30, 2016 at 2:20 PM, Roman <rsht...@yahoo.com.invalid> > > > wrote: > > > > > > > > > Raul, > > > > > > > > > > Small comment from me. > > > > > > > > > > >* As a Flink sink => inject data directly into a cache via a > > > > DataStreamer. > > > > > After reviews, IGNITE-813 is exactly this functionality. > > > > > > > > > > > > > > That's cool, Roman! The idea would be to host these (richer) modules > as > > > > Flink connectors, like they do with others: > > > > > > > > > https://github.com/apache/flink/tree/master/flink-streaming-connectors > > > > https://github.com/apache/flink/tree/master/flink-batch-connectors > > > > > > > > > >
