Re: Information regarding RichParallelSourceFunction

2016-12-25 Thread Saikat Maitra
Hi Robert Yes, I intend to commit the Ignite Source module as part of this jira ticket. https://issues.apache.org/jira/browse/IGNITE-3303 I am trying to resolve an issue specific to cancelling the Source function. Flink serializes and distributes source function and I can see in debugger differe

Re: Information regarding RichParallelSourceFunction

2016-12-24 Thread Robert Metzger
Hi Saikat, there is already a connector for Ignite and Flink in the Apache Ignite project: https://github.com/apache/ignite/tree/master/modules/flink Maybe you can contribute your Ignite source to that project as well. Regards, Robert On Thu, Dec 22, 2016 at 10:04 AM, Fabian Hueske wrote: > H

Re: Information regarding RichParallelSourceFunction

2016-12-22 Thread Fabian Hueske
Hi, Flink serializes all user functions (including source functions) with Java Serialization to ship them to the worker processes. That's also why everything in a user function must be Serializable. There is not an easy way to synchronize running tasks. Each task has its own function object and t

Information regarding RichParallelSourceFunction

2016-12-22 Thread Saikat Maitra
Hello, I am working on building Apache Ignite connector for Apache flink. I am currently developing the SourceFunction to consume Cache event from Ignite cluster. Here is the PR https://github.com/apache/ignite/pull/870/files I am observing that during unit tests the IgniteSource instances are d