that is nice. So, only by this Flink shuffles fewer data them Spark. Now I need to plug Prometheus and Grafana to show it.
Thanks Yun for your help! *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Fri, Oct 11, 2019 at 12:08 PM Yun Tang <myas...@live.com> wrote: > Hi Felipe > > From what I remember, Spark still use micro-batch to shuffle data in > structed streaming. > For Flink, it actually process elements per record, there is no actual > disk-io shuffle in Flink streaming. And record would emit to downstream by > select specific channel through network[1]. That's why we need to call " > keyBy" before using windows, "KeyGroupStreamPartitioner" would then be > used to select the target channel based on the key group index. Data would > first be stored in local state backend and wait for polled out once a > window triggered but not "shuffled" until a window triggered. > > > [1] > https://github.com/apache/flink/blob/2c2095bdad3d47f27973a585112ed820f457de6f/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ChannelSelectorRecordWriter.java#L60 > > Best > Yun Tang > > ------------------------------ > *From:* Felipe Gutierrez <felipe.o.gutier...@gmail.com> > *Sent:* Friday, October 11, 2019 15:47 > *To:* Yun Tang <myas...@live.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Difference between windows in Spark and Flink > > Hi Yun, > > that is a very complete answer. Thanks! > > I was also wondering about the mini-batches that Spark creates when we > have to create a SparkStream context. It still remains for all versions of > stream processing in Spark, isn't it? And because that I Spark shuffles > data [1] to wide-dependent operators every time mini-batch ends, doesn't it? > In this way Flink does not have mini-batches, hence I will shuffle data to > wide-dependent operators only when a window is triggered. Am I correct? > > [1] > http://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/rdd-programming-guide.html#shuffle-operations > > Thanks, > Felipe > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez * > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > > > On Thu, Oct 10, 2019 at 7:25 PM Yun Tang <myas...@live.com> wrote: > > Hi Felipe > > Generally speaking, the key difference which impacts the performance is > where they store data within windows. > For Flink, it would store data and its related time-stamp within windows > in state backend[1]. Once window is triggered, it would pull all the stored > timer with coupled record-key, and then use the record-key to query state > backend for next actions. > > For Spark, first of all, we would talk about structured streaming [2] as > it's better than previous spark streaming especially on window scenario. > Unlike Flink built-in supported rocksDB state backend, Spark has only one > implementation of state store providers. It's HDFSBackedStateStoreProvider > which stores all of the data in memory, what is a very memory consuming > approach and might come across OOM errors[3][4][5]. > > To avoid this, Databricks Runtime offer a 'RocksDBStateStoreProvider' but > not open-source. We're lucky that open-source Flink already offers built-in > RocksDB state backend to avoid OOM problem. Moreover, Flink community > recently are developing spill-able memory state backend [7]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html > [2] > https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time > [3] > https://medium.com/@chandanbaranwal/state-management-in-spark-structured-streaming-aaa87b6c9d31 > [4] > http://apache-spark-user-list.1001560.n3.nabble.com/use-rocksdb-for-spark-structured-streaming-SSS-td34776.html#a34779 > [5] https://github.com/chermenin/spark-states > [6] > https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-of-stateful-streaming-queries > [7] https://issues.apache.org/jira/browse/FLINK-12692 > > Best > Yun Tang > > > > ------------------------------ > *From:* Felipe Gutierrez <felipe.o.gutier...@gmail.com> > *Sent:* Thursday, October 10, 2019 20:39 > *To:* user <user@flink.apache.org> > *Subject:* Difference between windows in Spark and Flink > > Hi all, > > I am trying to think about the essential differences between operators in > Flink and Spark. Especially when I am using Keyed Windows then a reduce > operation. > In Flink we develop an application that can logically separate these two > operators. Thus after a keyed window I can use > .reduce/aggregate/fold/apply() functions [1]. > In Spark we have window/reduceByKeyAndWindow functions which to me appears > it is less flexible in the options to use with a keyed window operation [2]. > Moreover, when these two applications are deployed in a Flink and Spark > cluster respectively, what are the differences between their physical > operators running in the cluster? > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#windows > [2] > https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations > > Thanks, > Felipe > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez * > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > >
