This is the issue where .sortByKey() launches a cluster job when it shouldn't because it's a transformation not an action.
https://issues.apache.org/jira/browse/SPARK-1021 I'd appreciate a fix too but don't currently have any thoughts on how to proceed forward. Andrew On Thu, May 8, 2014 at 2:16 PM, Mark Hamstra <m...@clearstorydata.com>wrote: > I'm trying to decide whether attacking the underlying issue of > RangePartitioner running eager jobs in rangeBounds (i.e. SPARK-1021) is a > better option than a messy workaround for some async job-handling stuff > that I am working on. It looks like there have been a couple of aborted > attempts to solve the problem, but no solution or clear path to one at this > point. > > Has anybody made any further progress or does anyone have any new ideas on > how to proceed? >
