Re: Aggregating over sorted data

2016-12-22 Thread Koert Kuipers
without any more changes to current Spark implementation, I think > >> this > >> is the one possible way to achieve the required function to aggregate on > >> sorted data per key. > >> > >> > >> > >> > >> > >> - > >>

Re: Aggregating over sorted data

2016-12-22 Thread trsell
t; > I was hoping to find something like: Efficient sortByKey to work with… > > > > *From:* Koert Kuipers [via Apache Spark Developers List] > [mailto:ml-node+[hidden > email] <http:///user/SendEmail.jtp?type=node&node=20334&i=0>] > *Sent:* Thursday, Decembe

RE: Aggregating over sorted data

2016-12-22 Thread assaf.mendelson
+s1001551n20332...@n3.nabble.com] Sent: Thursday, December 22, 2016 7:14 AM To: Mendelson, Assaf Subject: Re: Aggregating over sorted data it can also be done with repartition + sortWithinPartitions + mapPartitions. perhaps not as convenient but it does not rely on undocumented behavior. i used this approach

Re: Aggregating over sorted data

2016-12-22 Thread Liang-Chi Hsieh
sorted data per key. >> >> >> >> >> >> - >> Liang-Chi Hsieh | @viirya >> Spark Technology Center >> http://www.spark.tc/ >> -- >> View this message in context: http://apache-spark- >> develop

Re: Aggregating over sorted data

2016-12-21 Thread Koert Kuipers
is > is the one possible way to achieve the required function to aggregate on > sorted data per key. > > > > > > - > Liang-Chi Hsieh | @viirya > Spark Technology Center > http://www.spark.tc/ > -- > View this message in context: http://apache-spark- &g

Re: Aggregating over sorted data

2016-12-21 Thread Liang-Chi Hsieh
. - Liang-Chi Hsieh | @viirya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Aggregating-over-sorted-data-tp1p20331.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: Aggregating over sorted data

2016-12-21 Thread Koert Kuipers
rya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark- developers-list.1001551.n3.nabble.com/Aggregating-over- sorted-data-tp1p20310.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Aggregating over sorted data

2016-12-20 Thread Liang-Chi Hsieh
te on sorted data per key. - Liang-Chi Hsieh | @viirya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Aggregating-over-sorted-data-tp1p20310.html Sent from the Apache Spark Developers List mailing list

Re: Aggregating over sorted data

2016-12-19 Thread Koert Kuipers
s feature. > > > > > > > - > Liang-Chi Hsieh | @viirya > Spark Technology Center > http://www.spark.tc/ > -- > View this message in context: http://apache-spark- developers-list.1001551.n3.nabble.com/A

Re: Aggregating over sorted data

2016-12-19 Thread Robin East
t; http://www.spark.tc/ > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Aggregating-over-sorted-data-tp1p20273.html > Sent from the Apache Spark Developers List mailing list archive at Nabble.com. > >

Re: Aggregating over sorted data

2016-12-18 Thread Liang-Chi Hsieh
park Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Aggregating-over-sorted-data-tp1p20273.html Sent from the Apache Spark Developers List mailing list archiv

Re: Aggregating over sorted data

2016-12-12 Thread nsyca
ce the processing in this case? - Nattavut Sutyanyong | @nsyca Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Aggregating-over-sorted-data-tp1p20206.html Sent from the Apache Spark Developers List m

Aggregating over sorted data

2016-11-23 Thread assaf.mendelson
hould be a feature to add (I can open a JIRA if people think it is a good idea). Assaf. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Aggregating-over-sorted-data-tp1.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.