Hi all, I've been reading Ignite docs but I'm not getting something.
It looks to me that the streaming processing API is rather simple, and even hacky. For instance, Ignite supports stream windows only as a consequence of having eviction policies on caches, but it lacks many other features that we generally see on other stream processing frameworks such as Flink. For instance, sliding windows (same length, different beginning time [1]), event time windows (times defined by the data itself, not processing time [2]) and different transformations (filter, fold, reduce, map, etc [3]). I may be able to do some of this using things like the eviction policy to define the length of a window, but isn't it a better idea to use Ignite as an in-memory data storage, and a fully fledged stream processing framework on top of it to define the transformations to apply to the data? My initial idea -which I haven't tried yet- is to use collocation to run a closure where the data resides (affinity call/run), and use that closure to execute a Flink pipeline (job) locally on that node, then using a custom made data source I should be able to plug the data from the local Ignite cache to the Flink pipeline and back into a cache using an Ignite sink. That would imply running the Flink job locally and completely disable its own job distribution support (that way I save time and bandwidth since the job is executed only on one node: the node which owns the data). Is it what you would/usually do to process Ignite data as a stream? Any additional insight regarding stream processing on Ignite is very welcome! Best regards, Matt [1] https://ci.apache.org/projects/flink/flink-docs-release- 1.2/dev/windows.html#sliding-windows [2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/ dev/event_time.html [3] https://ci.apache.org/projects/flink/flink-docs-release- 1.2/dev/datastream_api.html#datastream-transformations
