Re: [discuss] making SparkEnv private in Spark 2.0

2016-03-20 Thread Reynold Xin
On Wed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan wrote: > b) Shuffle manager (to get shuffle reader) > What's the use case for shuffle manager/reader? This seems like using super internal APIs in applications.

Re: [discuss] making SparkEnv private in Spark 2.0

2016-03-19 Thread Mridul Muralidharan
We have custom join's that leverage it. It is used to get to direct shuffle'ed iterator - without needing sort/aggregate/etc. IIRC the only way to get to it from ShuffleHandle is via shuffle manager. Regards, Mridul On Wed, Mar 16, 2016 at 3:36 PM, Reynold Xin wrote: > > On Wed, Mar 16, 2016 a

Re: [discuss] making SparkEnv private in Spark 2.0

2016-03-19 Thread Mridul Muralidharan
We use it in executors to get to : a) spark conf (for getting to hadoop config in map doing custom writing of side-files) b) Shuffle manager (to get shuffle reader) Not sure if there are alternative ways to get to these. Regards, Mridul On Wed, Mar 16, 2016 at 2:52 PM, Reynold Xin wrote: > Any