Hey! Both groupBy and partitionBy will trigger a shuffle over the network based on some key, assuring that elements with the same keys end up on the same downstream processing operator.
The difference between the two is that groupBy in addition to this returns a GroupedDataStream which lets you execute some special operations, such as key based rolling aggregates. PartitionBy is useful when you are using simple operators but still want to control the messages received by parallel instances (in a mapper for example). Cheers, Gyula tambunanw <if05...@gmail.com> ezt írta (időpont: 2015. júl. 3., P, 10:32): > Hi All, > > I'm trying to digest what's the difference between this two. From my > experience in Spark GroupBy will cause shuffling on the network. Is that > the > same case in Flink ? > > I've watch videos and read a couple docs about Flink that's actually Flink > will compile the user code into it's own optimized graph structure so i > think Flink engine will take care of this one ? > > From the docs for Partitioning > > > http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#partitioning > > Is that true that GroupBy is more advanced than PartitionBy ? Can someone > elaborate ? > > I think this one is really confusing for me that come from Spark world. Any > help would be really appreciated. > > Cheers > > > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Streaming-PartitionBy-vs-GroupBy-differences-tp1927.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >
