Re: Apache Flink <=> Apache Ignite integration

2016-07-19 Thread Anton Vinogradov
Hi All, I'll review it in the near future. On Tue, Jul 19, 2016 at 11:53 AM, Denis Magda wrote: > Hi Saikat, > > Thanks for this contribution. > > *Anton V.*, please review the following contribution. > > — > Denis > > On Jul 16, 2016, at 11:09 PM, Saikat Maitra > wrote: > > Hi > > I have raise

Re: Apache Flink <=> Apache Ignite integration

2016-07-19 Thread Denis Magda
Hi Saikat, Thanks for this contribution. Anton V., please review the following contribution. — Denis > On Jul 16, 2016, at 11:09 PM, Saikat Maitra wrote: > > Hi > > I have raised a PR for the following scope. > > As a Flink source => run a continuous query against one or multiple > caches >

Re: Apache Flink <=> Apache Ignite integration

2016-07-16 Thread Saikat Maitra
Hi I have raised a PR for the following scope. As a Flink source => run a continuous query against one or multiple caches PR https://github.com/apache/ignite/pull/870 Jira https://issues.apache.org/jira/browse/IGNITE-3303 Please review and share feedback. Regards Saikat On Mon, Apr 4, 2016 a

Re: Apache Flink <=> Apache Ignite integration

2016-04-05 Thread Stephan Ewen
Hi Raul! Concerning the source connector and position marker: Great idea! The FlinkKafkaConsumer uses pretty much the same trick - the offset-per-partition is used to filter during replays. Greetings, Stephan On Tue, Apr 5, 2016 at 2:24 AM, Raul Kripalani wrote: > On Mon, Apr 4, 2016 at 3:54

Re: Apache Flink <=> Apache Ignite integration

2016-04-04 Thread Raul Kripalani
On Mon, Apr 4, 2016 at 3:54 PM, Stephan Ewen wrote: > > - Sounds like the having Ignite for snapshots should work pretty much > out > of the box (via the IGFS) > - The source and sink connector sounds like the next logical step. Does > Ignite have a notion of stream partitions and offsets, to

Re: Apache Flink <=> Apache Ignite integration

2016-04-04 Thread Raul Kripalani
On Mon, Apr 4, 2016 at 3:35 PM, Robert Metzger wrote: > thanks a lot for reaching out to the Flink community. > I'm really excited to see a Flink connector in Ignite. If you feel that > the > connector would be more suitable for our "connector library" feel free to > open a JIRA and open a pull r

Re: Apache Flink <=> Apache Ignite integration

2016-04-04 Thread Stephan Ewen
Hi! - Sounds like the having Ignite for snapshots should work pretty much out of the box (via the IGFS) - The source and sink connector sounds like the next logical step. Does Ignite have a notion of stream partitions and offsets, to build a consistent replay around? This should probably have

Re: Apache Flink <=> Apache Ignite integration

2016-04-04 Thread Robert Metzger
Hi Raul, thanks a lot for reaching out to the Flink community. I'm really excited to see a Flink connector in Ignite. If you feel that the connector would be more suitable for our "connector library" feel free to open a JIRA and open a pull request. Were there requests in the Ignite community to

Re: Apache Flink <=> Apache Ignite integration

2016-03-31 Thread Saikat Maitra
Hi , I agree with Roman and Raul. https://issues.apache.org/jira/browse/IGNITE-813 allows injecting data to into cache via Data Streamer. Integrating with Ignite FileSystem for source and sink will allow for bidirectional connector. It will also allow easier implementation for DataStream transform

Re: Apache Flink <=> Apache Ignite integration

2016-03-31 Thread Aljoscha Krettek
Hi, it should already be possible to use the Ignite FileSystem to store state since we just use the HDFS FileSystem interface for that. Of course, one would have to properly set up the jars and paths and everything for Flink to pick up the IGFS classes. Cheers, Aljoscha On Wed, 30 Mar 2016 at 16:

Re: Apache Flink <=> Apache Ignite integration

2016-03-30 Thread Roman
Raul, Small comment from me. >* As a Flink sink => inject data directly into a cache via a DataStreamer. After reviews, IGNITE-813 is exactly this functionality. -Roman

Re: Apache Flink <=> Apache Ignite integration

2016-03-30 Thread Raul Kripalani
On Wed, Mar 30, 2016 at 2:20 PM, Roman wrote: > Raul, > > Small comment from me. > > >* As a Flink sink => inject data directly into a cache via a DataStreamer. > After reviews, IGNITE-813 is exactly this functionality. > > That's cool, Roman! The idea would be to host these (richer) modules as F

Apache Flink <=> Apache Ignite integration

2016-03-30 Thread Raul Kripalani
Hello from the Apache Ignite community! Last year there was an interesting thread [1] about such integration. Unfortunately there's been little follow-through, so let's try and fix that in 2016 ;-) I'm sure a lot has changed in the Flink community, with the recent graduation and 1.0 release, so I