Hi! I think sort partition is the right think, if you have only one partition (which makes sense, if you want a total order). It is not a parallel operation any mode, so use it only after the data size has been reduced (filters / aggregations).
What about "data.sortPartition().setParallelism(1)". Does that work for you? Greetings, Stephan On Sun, Mar 15, 2015 at 4:47 PM, Kristoffer Sjögren <[email protected]> wrote: > Thanks for your answer. I guess i'm a bit infected by writing to much > Crunch code and I also suspected that getDataSet() was the wrong thing to > do :-) > > However I was expecting DataSet.sortPartition to do the sorting, but this > method is missing in 0.8.1? > > Do you have a minimal example? I was looking through the tests but most of > them use sortPartition. > > Cheers, > -Kristoffer > > > > On Sun, Mar 15, 2015 at 4:22 PM, Stephan Ewen <[email protected]> wrote: > >> Hi Kristoffer! >> >> There are a few issues with that code: >> >> 1) Grouping and then calling "sort group" sorts within the group. In your >> case, you group after the entire element and each group has on value - the >> element. Sorting inside the group does not make any difference. There is no >> order across groups. >> >> 2) This code never groups and sorts. The calls to "groupBy(0).sortGroup(0, >> Order.DESCENDING)." do not group and sort already, they set up a grouping >> to be used with a reduce or aggregate function. The "getDataSet()" call >> gets you the original data set, which is the original input. >> >> To see an illustration of this, get the program plan >> (env.getExecutionPlan()). You can render it using the html file >> "tools/planVisualizer.html". >> >> Greetings, >> Stephan >> >> >> On Sun, Mar 15, 2015 at 3:29 PM, Kristoffer Sjögren <[email protected]> >> wrote: >> >>> Hi >>> >>> This is silly but I can't understand why the following code doesn't sort >>> the collection of integers. It seems to be reasonable thing to do from an >>> API perspective? >>> >>> Cheers, >>> -Kristoffer >>> >>> final ExecutionEnvironment env = >>> ExecutionEnvironment.getExecutionEnvironment(); >>> env.fromCollection(Lists.newArrayList(2,1,5,3,4,5)).map(new >>> MapFunction<Integer, Tuple1<Integer>>() { >>> @Override >>> public Tuple1<Integer> map(Integer value) throws Exception { >>> return new Tuple1(value); >>> } >>> }).groupBy(0).sortGroup(0, Order.DESCENDING).getDataSet().print(); >>> env.execute(); >>> >>> >>> >> >
