Hi everyone,
I wonder if there are plans to implement real time streaming in Spark. I see that in Spark 2.0 Trigger can have more implementations than ProcessingTime. In my opinion Real Time streaming (so reaction on every event - like continous queries in Apache Ignite) will be very useful and will fill gap that is currently in Spark. Now, if we must implement both real-time streaming and batch jobs, the streaming must be done in other frameworks as Spark allows us only to process event in Micro Batches. Matei Zaharia wrote in Databricks blog about Continuous Applications [1], in my opinion adding EventTrigger will be next big step to Continuous Applications. What do you think about it? Are there any plans to implement such event-based trigger? Of course I can help with implementation, however I'm just starting learning Spark internals and it will take a while before I would be able to write something. Pozdrawiam / Best regards, Tomek [1] https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html [https://databricks.com/wp-content/uploads/2016/07/spark-2-continuous-apps-OG.png]<https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html> Continuous Applications: Evolving Streaming in Apache Spark 2.0<https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html> databricks.com Apache Spark 2.0 lays the foundation for Continuous Applications, a simplified and unified way to write end-to-end streaming applications that reacts to data in real-time.
